From 69baf93f903375928496da0e507b355d26fe629d Mon Sep 17 00:00:00 2001 From: Wolfgang Bangerth Date: Fri, 28 May 2021 12:47:34 -0600 Subject: [PATCH] Go through the rest of the paper. --- 9.3/paper.tex | 248 +++++++++++++++++++++++++------------------------- 1 file changed, 125 insertions(+), 123 deletions(-) diff --git a/9.3/paper.tex b/9.3/paper.tex index 11aa7e2..b7d8b2d 100644 --- a/9.3/paper.tex +++ b/9.3/paper.tex @@ -694,17 +694,24 @@ supported. \subsection{Advances in the particle infrastructure} \label{subsec:particles} -TODO +\todo[inline]{TODO} + %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% \subsection{Advances in the multigrid infrastructure} \label{subsec:mg} -Until now, \dealii has only supported local smoothing multigrid -algorithms~\citep{ClevengerHeisterKanschatKronbichler2019}; in local -smoothing algorithms, smoothers only act on the cells of a given refinement level, -skipping those parts of the mesh that are not adaptively refined to that level. -The current release now also has support for global coarsening~\citep{becker2000multigrid, sundar2012parallel} when +Until now, \dealii has only supported ``local smoothing'' multigrid +algorithms~\citep{ClevengerHeisterKanschatKronbichler2019} wherein +smoothers only act on the cells of a given refinement level, +skipping those parts of the mesh that have not been adaptively refined +to that level. This approach guarantees that the work done summed up +over all levels is proportional to the number of unknowns, and is +consequently necessary so that the overall multigrid preconditioner +can have a complexity of ${\cal O}(N)$. + +In contrast, the current release now also supports ``global +coarsening'' algorithms \citep{becker2000multigrid, sundar2012parallel} when using continuous (\texttt{FE\_Q}, \texttt{FE\_SimplexP}) and discontinuous (\texttt{FE\_DGQ}, \texttt{FE\_SimplexDGP}) elements. Global coarsening builds multigrid levels for the entire @@ -732,7 +739,8 @@ hanging nodes within each level and general require more computational work per iteration overall. The transfer operators between two levels have been implemented in the new class \texttt{MGTwoLevel\allowbreak Transfer}, which can be set up via the functions \texttt{MGTwoLevel\allowbreak Transfer::\allowbreak reinit\_\allowbreak geometric\_\allowbreak transfer()} or \texttt{MGTwo\allowbreak LevelTransfer::\allowbreak reinit\_\allowbreak polynomial\_\allowbreak transfer()} for given -\texttt{DoFHandler} and \texttt{AffineConstraint} classes of two levels. The resulting transfer operators +\texttt{DoFHandler} and \texttt{AffineConstraint} objects +corresponding to the two levels. The resulting transfer operators can then be collected in a single \texttt{MGTransfer\allowbreak GlobalCoarsening} object that can be used just as the previous workhorse \texttt{MGTransferMatrixFree} within the \texttt{Multigrid} algorithm. @@ -752,9 +760,15 @@ functions) in the context of a hybrid multigrid algorithm \subsection{Advances in the matrix-free infrastructure} \label{subsec:mf} +\dealii{}'s matrix-free framework enables high-throughput operations +for applications in which only the availability of the \textit{action} +of a matrix, but not the entries of the matrix, are necessary. This +framework has been substantially extended in the current release. + + \subsubsection{Precompilation of evaluation kernels} -The classes \texttt{FEEvaluation} and \texttt{FEFaceEvaluation} use template parameters for +The \texttt{FEEvaluation} and \texttt{FEFaceEvaluation} classes use template parameters for the polynomial degree of the finite element $k$ and the number of the 1D quadrature points $q$ to generate near-optimal code for these operations. For application codes that rely on operators of many different degrees (e.g., because @@ -763,7 +777,7 @@ can be overly complex and incur long compile times. In the current release, specializations of these classes that do not rely on the template parameters $k$ and $q$ (expressed in the code -with ``-1'' and ``0'') have been added. For example: +using special values ``-1'' and ``0'') have been added. For example: \begin{c++} FEEvaluation phi(range, dofhandler_index, quadrature_index, first_selected_component); @@ -791,11 +805,17 @@ DEAL_II_NAMESPACE_CLOSE \subsubsection{Parallel matrix-free $hp$-implementation}\label{subsubsection:mf:hp} -With release 9.1 \cite{dealII91}, large parts of the $hp$-algorithms in \dealii were parallelized, enabling -parallel matrix-based simulations with the $hp$-infrastructure. In the present +With release 9.1, large parts of the $hp$-algorithms +in \dealii were ported to a model that allows for +parallel matrix-based simulations \cite{dealII91}. In the present release, the parallel $hp$ support was extended to \texttt{MatrixFree}. -Until now, the \texttt{FEEvaluation} classes used the template parameters $k$ and $q$ to select the correct active FE and quadrature index and cumbersome detection of subranges of the same degree. +Until now, the \texttt{FEEvaluation} classes used the template +parameters $k$ and $q$ to select the correct active FE and quadrature +index and cumbersome detection of subranges of the same degree. +\todo{I don't understand what ``detection of subranges of the same + degree'' refers to. Please clarify. In particular, I don't know what +``subranges'' mean in this paragraph.} The creation of subranges is now performed internally, and the non-templated versions of the \texttt{FEEvaluation} classes have been extended for the $hp$-case. To determine the desired FE and quadrature index of a subrange, the current cell/face range has to be provided @@ -808,18 +828,18 @@ in the context of a hybrid-multigrid solver. \subsection{MPI-3.0 shared-memory support}\label{subsec:sm} -In many large computations, certain pieces of data are computed once -and then treated as read-only. If this information is needed by more +In many large computations, certain pieces of data are computed +(or read from disk) once and then treated as read-only. If this information is needed by more than one MPI process, it is more efficient to store this information only once in shared memory among all processes located on a multicore node. MPI supports the creation of such shared memory windows since version 3.0, and deal.II can now use this in the \texttt{AlignedVector} and \texttt{Table} classes that are often used -for large lookup tables. +for large lookup tables for classes such as \texttt{InterpolatedTensorProductGridData}. -The feature is also used in the \texttt{MatrixFree} and +Shared memory storage is also used in the \texttt{MatrixFree} and \texttt{LinearAlgebra::\allowbreak distributed::\allowbreak Vector} classes. If \texttt{MatrixFree} has been configured by setting -\texttt{MatrixFree::AdditionalData::communicator\_sm} appropriately, then +\texttt{MatrixFree::\allowbreak AdditionalData::\allowbreak communicator\_sm} appropriately, then \texttt{MatrixFree::create\_dof\_vector()} creates vectors that share information among all processes on one node. As a consequence, the \texttt{FEEvaluation} classes can access vector elements owned by @@ -839,14 +859,18 @@ For more details and the usage of the feature in the library \texttt{hyper.deal} \subsection{Evaluation and integration at arbitrary points} \label{subsec:fepointvalues} -In a number of circumstances, finite element solutions need to be evaluated on +In a number of circumstances, finite element solutions need to be evaluated at arbitrary reference points that change from one element to next. Two important examples are particle simulations coupled to a finite element solution, or algorithms on non-matching grids. The existing \texttt{FEValues} class is a poor -fit for this task, as it relies on tabulating all information before its usage, -necessitating a separate setup for every cell. The new class -\texttt{FEPointEvaluation} provides a more convenient interface for this -task. For tensor product finite elements (\texttt{FE\_Q}, \texttt{FE\_DGQ}) +fit for this task, as it is based on the assumption that evaluation of +shape functions and their derivatives happens at the same quadrature +points on every cell, and that consequently expensive computations can +be done once and results re-used for many subsequent cells. The new class +\texttt{FEPointEvaluation} provides a more convenient interface for +cases where the evaluation on different cells does not happen at the +same points mapped from the reference cell. +For tensor product finite elements (\texttt{FE\_Q}, \texttt{FE\_DGQ}) and tensor product mappings (\texttt{MappingQGeneric} and derived classes), the new approach is also very fast, as it can use some of the matrix-free infrastructure and vectorization facilities. @@ -854,11 +878,13 @@ infrastructure and vectorization facilities. To give an example of the new features, let us consider the evaluation of a surface tension force in the context of sharp-interface methods, whose contribution is added to a fluid solver by multiplication with test function -and addition over quadrature points: +and addition over quadrature points located at the interface section +$\Gamma_K=\Gamma \cup K$ located on the current cell $K$ and that is, +in general, located differently within $K$ than for any other cell: \begin{align*} -\left(\vec{v}, \kappa \, \vec{n}\right)_\Gamma +\left(\mathbf{v}, \kappa \, \mathbf{n}\right)_\Gamma \approx -\sum_q\left(\vec{v}, \kappa(\vec{p}_q) \, \vec{n}(\vec{p}_q) (JxW)_q\right). +\sum_q \mathbf{v}(\mathbf{x}_q) \cdot \left(\kappa(\mathbf{x}_q) \, \mathbf{n}(\mathbf{x}_q)\right) (JxW)_q. \end{align*} In \dealii{}, this can now be conveniently written as \begin{c++} @@ -870,21 +896,24 @@ for (unsigned int q = 0; q < n_points; ++q) phi_normal.get_value(q) * JxW[q], q); phi_force.integrate(force_values, EvaluationFlags::values); \end{c++} -The quadrature points (at reference positions - \texttt{reference\_points}) and the related \texttt{JxW} value can, e.g., come from -a mesh of codimension one. Determining to which \texttt{cell} a quadrature -point belongs to on the background mesh, including the reference-cell -coordinates \texttt{reference\_points}, can be determined with functions like +The quadrature points (at reference positions +\texttt{reference\_points}) and the related \texttt{JxW} value can, +for example, come from +a mesh of lower dimension. Determining to which \texttt{cell} a quadrature +point belongs to on the background mesh, including the reference cell +coordinates \texttt{reference\_points}, is aided by functions like \texttt{GridTools::find\_all\_active\_cells\_around\_point()}. While these functions have been available in \dealii{} previously, their performance has been considerably enhanced with the aforementioned more optimized code paths for selected mappings. While \texttt{FEPointEvaluation} assumes that evaluation points are already sorted according to -the owning cells and such can concentrate on cell-local operations, the new class -\texttt{RemovePointEvaluation} is responsible to determine the owning cells in a distributed +the owning cells and thus can focus on cell-local operations, the new class +\texttt{RemotePointEvaluation} is responsible for determine the owning cells in a distributed context and for providing efficient communication patterns for the data exchange. In \texttt{deal.II}, the class has been successfully applied together with \texttt{FEPointEvaluation} to evaluate a distributed -solution vector at arbitrary points (see \texttt{VectorTools::evaluate\_at\_points()}). +solution vector at arbitrary points (see also the +\texttt{VectorTools::evaluate\_at\_points()} function). %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% @@ -892,17 +921,15 @@ solution vector at arbitrary points (see \texttt{VectorTools::evaluate\_at\_poin \label{subsec:steps} Many of the \dealii{} tutorial programs were revised in a variety of -ways as part of this release. TODO - -In addition, there are a number of new tutorial programs: +ways as part of this release. In addition, there are a number of new tutorial programs: \begin{itemize} \item \texttt{step-19} is an introductory demonstration of \dealii{}'s particle functionality. It solves the coupled problem of charged particles and an electric field, using a cathode tube as an example. -\item \texttt{step-68} TODO - +\item \texttt{step-68} \todo[inline]{TODO} + \item \texttt{step-71} focuses on automatic and symbolic differentiation (AD and SD, in short) as a tool to make solvers for complex, nonlinear problems possible. To this end, \dealii{} can interface to a number @@ -936,12 +963,12 @@ In addition, there are a number of new tutorial programs: compressible Euler equations discretized with a high-order discontinuous Galerkin (DG) scheme, using the matrix-free infrastructure just as \texttt{step-67} does. The tutorial presents advanced topics, like the usage of cell-centric loops and - the new MPI-3.0 shared-memory capabilities of \texttt{MatrixFree} tor each high + the new MPI-3.0 shared-memory capabilities of \texttt{MatrixFree} to reach high throughput. Furthermore, the utilization of the template parameter - \texttt{VectorizedArrayType} and the application of lambdas to capture cell and face + \texttt{VectorizedArrayType} and the application of lambda functions to describe cell and face integrals are discussed. -\item \texttt{step-77} is a program that illustrate \texttt{dealii}'s +\item \texttt{step-77} is a program that illustrate \dealii{}'s interfaces to the SUNDIALS library \cite{sundials}, and specifically the KINSOL nonlinear solver. Like the \texttt{step-72} program mentioned above, it is a variation of the minimal surface solver @@ -954,11 +981,11 @@ In addition, there are a number of new tutorial programs: wheel but instead building on an existing and well-tuned software such as KINSOL. - \dealii{}'s interfaces were also updated to the latest SUNDIALS release, 5.7. + \dealii{}'s interfaces to the various SUNDIALS sub-packages were also updated to the latest SUNDIALS release, 5.7. -\item \texttt{step-78} TODO +\item \texttt{step-78} \todo[inline]{TODO} -\item \texttt{step-79} TODO +\item \texttt{step-79} \todo[inline]{TODO} \end{itemize} There is also a new program in the code gallery (a collection of @@ -966,7 +993,7 @@ user-contributed programs that often solve more complicated problems than tutorial programs, and intended as starting points for further research rather than as teaching tools): \begin{itemize} - \item preCICE: TODO + \item preCICE: \todo[inline]{TODO} \end{itemize} @@ -989,9 +1016,11 @@ may have been more widely used: and classes like \texttt{DataOut} or \texttt{SolutionTransfer} for example no longer require them. If you rely on these template arguments, an interim namespace \texttt{Legacy} has been introduced that provides all affected - classes with the old interface for the transition phase. + classes with the old interface for a transition period. \end{itemize} +\todo[inline]{Check whether there are other things worth listing.} + %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% @@ -1001,7 +1030,7 @@ may have been more widely used: In order to justify the work the developers of \dealii{} put into this software, we ask that papers using the library reference one of the -\dealii{} papers. This helps us justify the effort we put into it. +\dealii{} papers. This helps us justify the effort we put into this library. There are various ways to reference \dealii{}. To acknowledge the use of the current version of the library, \textbf{please reference the present @@ -1012,7 +1041,8 @@ see \end{center} The original \dealii{} paper containing an overview of its -architecture is \cite{BangerthHartmannKanschat2007}. If you rely on +architecture is \cite{BangerthHartmannKanschat2007}, and a more recent +publication documenting \dealii{}'s design decisions is available as \cite{dealII2020design}. If you rely on specific features of the library, please consider citing any of the following: \begin{multicols}{2} @@ -1021,20 +1051,20 @@ following: \item For geometric multigrid: \cite{Kanschat2004,JanssenKanschat2011,ClevengerHeisterKanschatKronbichler2019}; \item For distributed parallel computing: \cite{BangerthBursteddeHeisterKronbichler11}; \item For $hp$-adaptivity: \cite{BangerthKayserHerold2007}; - \item For partition-of-unity (PUM) and enrichment methods of the - finite element space: \cite{Davydov2016}; + \item For partition-of-unity (PUM) and finite element enrichment methods: + \cite{Davydov2016}; \item For matrix-free and fast assembly techniques: \cite{KronbichlerKormann2012,KronbichlerKormann2019}; \item For computations on lower-dimensional manifolds: \cite{DeSimoneHeltaiManigrasso2009}; \item For curved geometry representations and manifolds: \cite{HeltaiBangerthKronbichlerMola2019}; - \vfill\null \columnbreak \item For integration with CAD files and tools: \cite{HeltaiMola2015}; \item For boundary element computations: \cite{GiulianiMolaHeltai-2018-a}; - \item For \texttt{LinearOperator} and \texttt{PackagedOperation} facilities: + \item For the \texttt{LinearOperator} and + \texttt{Packaged\-Operation} facilities: \cite{MaierBardelloniHeltai-2016-a,MaierBardelloniHeltai-2016-b}; \item For uses of the \texttt{WorkStream} interface: \cite{TKB16}; @@ -1106,68 +1136,62 @@ contributed code to this release:\\ % we should make sure he's listed. % % 9.3: updated 5/08/2021 PM -Alexander Grayver, -Andrew Davis, +% sorted by WB 5/28 +Pasquale Africa, +Tyler Anderson, +Mathias Anselmann, Arpit Babbar, +Maximilian Bergbauer, +Nicolas Barnafi, Bruno Blais, -Bruno Turcksin, -Ce Qin, -Christoph Kammer, +Michele Bucelli, +Marcus Calhoun-Lopez, +Praveen Chandrashekar, Conrad Clevenger, -Dominic Soldner, -Daniel Arndt, -Daniel Garcia-Sanchez, -Daniel Paukner, David F. Castellanos, -David Wells, -Doug Shi-Dong, -David Schneider, +Andrew Davis, Elias Dejene, -Graham Harper, -Ignacio Tomas, -Ivan Fumagalli, -Jake Harmon, -Jean-Paul Pelteret, -Jiaqi Zhang, -Julian Roth, -Katharina Kormann, -Konrad Simon, -Krishnakumar Gopalakrishnan, -Laura Prieto Saavedra, -Lei Qiao, -Luca Heltai, -Magdalena Schreter, -Malhar Tidke, -Marc Fehling, -Marcus Calhoun-Lopez, -Martin Kronbichler , -Mathias Anselmann, -Matthias Maier, -Maximilian Bergbauer, +Niklas Fehn, Menno Fraters, -Michele Bucelli, +Ivan Fumagalli, +Daniel Garcia-Sanchez, +Rene Gassmoeller, Nicola Giuliani, -Nicolas Barnafi, -Niklas Fehn, -Nils Much, +Krishnakumar Gopalakrishnan, +Alexander Grayver, Olivier Guevremont, -Pasquale Africa, -Peter Munch, -Peter Westerbaan, +Jake Harmon, +Graham Harper, +Uwe K\"ocher, +Katharina Kormann, +Christoph Kammer, +Wenyu Lei, +Zhou Lei, Phillip Mobley, +Nils Much, Pratik Nayak, -Praveen Chandrashekar, -Rene Gassmoeller, +Toni El Geitani Nehme, +Justin O'Connor, +Daniel Paukner, +Sebastian Proell, +Lei Qiao, +Ce Qin, Reza Rastak, +Julian Roth, +Laura Prieto Saavedra, +Doug Shi-Dong, +David Schneider, +Magdalena Schreter, Richard Schussnig, -Sebastian Proell, +Konrad Simon, +Dominic Soldner, Simon Sticko, -Timo Heister, -Toni El Geitani Nehme, -Wenyu Lei, -Wolfgang Bangerth, -Zhou Lei +Malhar Tidke, +Ignacio Tomas, +Peter Westerbaan, +Jiaqi Zhang. +\todo[inline]{Remove everyone who may end up being a co-author.} Their contributions are much appreciated! @@ -1181,7 +1205,7 @@ D.~Arndt and B.~Turcksin: Research sponsored by the Laboratory Directed Research Development Program of Oak Ridge National Laboratory, managed by UT-Battelle, LLC, for the U. S. Department of Energy. -W.~Bangerth, T.~C.~Clevenger, and T.~Heister were partially +W.~Bangerth and T.~Heister were partially supported by the Computational Infrastructure in Geodynamics initiative (CIG), through the National Science Foundation under Award No.~EAR-1550901 and The @@ -1192,18 +1216,6 @@ W.~Bangerth was also partially supported by award OAC-1835673 as part of the Cyb program, DMS-1821210, and EAR-1925595. -%D.~Davydov was supported by the German Research Foundation (DFG), grant DA -%1664/2-1 and the Bayerisches Kompetenznetzwerk -%f\"ur Technisch-Wissenschaftliches Hoch- und H\"ochstleistungsrechnen -%(KONWIHR). - -B.~Blais was partially supported by the National Science and Engineering Research Council of Canada(NSERC) through the RGPIN-2020-04510 Discovery Grant - -T.~C.~Clevenger was also partially supported EAR-1925575 and OAC-2015848. - -A.~V.~Grayver was partially supported by the European Space Agency -Swarm DISC program. - Timo Heister was also partially supported by the National Science Foundation (NSF) Award DMS-2028346, OAC-2015848, EAR-1925575, and by Technical Data Analysis, Inc. through US Navy STTR Contract N68335-18-C-0011. @@ -1228,20 +1240,10 @@ NSF Award DMS-1912847. D.~Wells was supported by the National Science Foundation (NSF) through Grant DMS-1344962. -Z.~Wang was partially -supported by the National Science Foundation under award OAC-1835673. - The Interdisciplinary Center for Scientific Computing (IWR) at Heidelberg University has provided hosting services for the \dealii{} web page. - -The authors acknowledge the Texas Advanced Computing Center (TACC) at The -University of Texas at Austin for providing access to HPC resources that have -contributed to the research results reported within this paper. - -Clemson University is acknowledged for generous allotment of compute time on -the Palmetto cluster. - +\todo[inline]{Everyone update. Add support for any additional authors.} \bibliography{paper}{} \bibliographystyle{abbrv} -- 2.39.5