From: Wolfgang Bangerth Date: Thu, 14 May 2020 19:42:26 +0000 (-0600) Subject: Merge branch 'my-stuff' X-Git-Url: https://gitweb.dealii.org/cgi-bin/gitweb.cgi?a=commitdiff_plain;h=a327b5ed26fdecf4b686796481fee75c51313b9a;p=release-papers.git Merge branch 'my-stuff' --- a327b5ed26fdecf4b686796481fee75c51313b9a diff --cc 9.2/paper.tex index 27f6753,ded6b2d..1e064e0 --- a/9.2/paper.tex +++ b/9.2/paper.tex @@@ -14,10 -14,10 +14,12 @@@ \usepackage{graphicx} \usepackage{xspace} +%\renewcommand{\baselinestretch}{2.0} + \usepackage[normalem]{ulem} + \usepackage{todonotes} + \pgfplotsset{compat=1.9} \newcommand{\specialword}[1]{\texttt{#1}} @@@ -298,262 -217,71 +229,301 @@@ can be foun \href{https://dealii.org/developer/doxygen/deal.II/changes_between_9_0_1_and_9_1_0.html}{ in the file that lists all changes for this release}, see \cite{changes91}. - %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% - \subsection{bla1} - \label{subsec:bla1} - + +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% +\subsection{Improved large-scale performance} +\label{subsec:performance} + +Large-scale simulations with 304,128 cores have revealed bottlenecks in release +9.1 during initialization due to the usage of expensive collective operations +like \texttt{MPI\_Allgather()} and \texttt{MPI\_\allowbreak Alltoall()}, e.g., during the +pre-computation of the index ranges of all processes, which have been stored in an array. +This information is needed to set up +the \texttt{Utilities::MPI::Par\-ti\-ti\-oner} class. +In release 9.2, we have removed such arrays and have replaced +the \texttt{MPI\_Allgather}/\allowbreak\texttt{MPI\_\allowbreak Alltoall} +function calls by consensus algorithms~\cite{hoefler2010scalable}, which can be +found in the namespace \texttt{Utilities::\allowbreak MPI::\allowbreak ConsensusAlgorithms} (short: \texttt{CA}): now, only the locally relevant information about the index ranges is (re)computed when needed, using these algorithms. +We provide two flavors of the consensus algorithm: the two-step +approach \texttt{CA::PEX} and the \texttt{CA::NBX}, +which uses only point-to-point communications and a single \texttt{MPI\_IBarrier()}. + +%Consensus algorithms are algorithms dedicated to efficient dynamic-sparse +%communication patterns. In this context, the term ``dynamic-sparse'' means +%that by the time this function is called, the other processes do not know +%yet that they have to answer requests and +%each process only has to communicate with a small subset of processes of the +%MPI communicator. We provide two flavors of the consensus algorithm: the two-step +%approach \texttt{ConsensusAlgorithms::PEX} and the \texttt{ConsensusAlgorithms::NBX}, +%which uses only point-to-point communications and a single \texttt{MPI\_IBarrier()}. +%The class \texttt{ConsensusAlgorithms::Selector} selects one of the two previous +%algorithms, depending on the number of processes. + +Users can apply the new algorithms for their own dynamic-sparse problems by +providing a list of target +processes and pack/unpack routines either by implementing the interface +\texttt{CA::\allowbreak Process} or by providing \texttt{std::function} +objects to \texttt{CA::AnonymousProcess}. + +By replacing the collective communications during setup and removing the arrays +that contain information for each process (enabled by the application of consensus +algorithms and other modifications---a full list of modifications leading to this +improvement can be found online), we were able to significantly improve the setup +time for large-scale simulations and to solve a Poisson problem with multigrid +with 12T unknowns. +Figure~\ref{} compares the timings of a simulation (incl. setup) with the +previous release 9.1 and with the current release 9.2. +{\color{red}TODO[Peter/Martin] description of the results} + +The new code has been also applied to solve problems with adaptively refined +meshes with more than 4B unknowns. {\color{red}TODO[Timo]} + + +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% +\subsection{A new fully distributed triangulation class} +\label{subsec:pft} + +By release 9.1, all triangulation classes of \texttt{deal.II} have in common that the coarse grid is shared by +all processes and the actual mesh used for computations is constructed by repeated +refinement. However, this has its limitations in industrial applications where, often, the mesh comes +from an external mesh generator in the form of a file that already contains millions +or tens of millions of cells. For such configurations, applications might already +run out of main memory, while reading the complete mesh by each MPI process. + +The new class \texttt{parallel::fullydistributed::Triangulation} targets this issue +by distributing also the coarse grid. Such +a triangulation can be created by providing to each process a \texttt{Triangulation\-De\-scrip\-tion::Description} struct, containing +1) the relevant data to construct the local part of the coarse grid, 2) the +translation of the local coarse-cell IDs to globally unique IDs, 3) the hierarchy +of mesh refinements, and 4) the owner of the cells on the active mesh level as well +as on the multigrid levels. Once the triangulation is set up with this struct, no adaptive +changes to the mesh are allowed at the moment. + +%The \texttt{TriangulationDescription::Description} struct can be filled manually or +%by the utility functions from the \texttt{TriangulationDescription::Utilities} +%namespace. The function \texttt{create\_\allowbreak description\_\allowbreak +%from\_ \allowbreak triangulation()} can convert a base triangulation (partitioned +%serial \texttt{Tri\-angulation} and \texttt{parallel::distributed::Triangulation}) +%to such a struct. The advantage of this approach is that all known utility +%functions from the namespaces \texttt{GridIn} and \texttt{GridTools} can be used +%on the base triangulations before converting them to the structs. Since this +%function suffers from the same main memory problems as described above, we also +%provide the function \texttt{create\_description\_from\_triangulation\_in\_groups()}, +%which creates the structs only on the master process in a process group. These +%structs are filled one by one and are sent to the relevant processes once they are +%ready. A sensible process group size might contain all processes of one compute node. + + +The new fully distributed triangulation class works---in contrast to +\texttt{parallel::distributed::\allowbreak Tri\-an\-gu\-la\-tion}---not only for 2D- and 3D- but also for +1D-problems. It can be used in the context of geometric multigrid methods and +supports periodic boundary conditions. Furthermore, hanging nodes are supported. + +%We intend to extend the usability of the new triangulation class in regard of +%different aspects, e.g., I/O. In addition, we would like to enable adaptive mesh +%refinement, a feature of the other triangulation classes, which is very much +%appreciated by many users. For repartitioning, we plan to use an oracle approach +%known from \texttt{parallel::distributed::Triangulation}. Here, we would like to +%rely on a user-provided partitioner, which might also be a graph partitioner. + + +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% +\subsection{Advances of the SIMD capabilities and the matrix-free infrastructure} +\label{subsec:mf} + +%\begin{itemize} +%\item ECL +%\item VectorizedArrayType +%\end{itemize} + +The class \texttt{VectorizedArray} is a key ingredient for the high +node-level performance of the matrix-free algorithms in deal.II~\cite{KronbichlerKormann2012, KronbichlerKormann2019}. It is a wrapper +class around $n$ vector entries of type \texttt{Number} and delegates relevant +function calls to appropriate Intrinsics instructions. Up to release 9.1, the +vector length $n$ has been set at compile time of the library to the highest +possible value supported by the given processor architecture. + +The class \texttt{VectorizedArray} has been made more user-friendly by making +it compatible with the STL algorithms found in the header \texttt{}. +The length of the vector can now be queried by \texttt{VectorizedArray::size()} and its underlying number type by \texttt{VectorizedArray::value\_type}. +Furthermore, the \texttt{VectorizedArray} class supports range-based iterations over its entries and, i.a., the following +algorithms: \texttt{std::\allowbreak ad\-vance()}, \texttt{std::distance()}, and \texttt{std::max\_element()}. + +This class has been also extended with the second optional template argument +\texttt{VectorizedArray} with \texttt{size} being related to the +vector length, i.e., the number of lanes to be used and the instruction set to be +used. By default, the number is set to the highest value supported by the given +hardware. A full list of supported +vector lengths is presented in Table~\ref{tab:vectorizedarray}. + +All matrix-free related classes (like \texttt{MatrixFree} and \texttt{FEEvaluation}) +have been templated with the floating-point number type \texttt{Number} (e.g. \texttt{double} or \texttt{float}); the computations were performed implicitly +on \texttt{VectorizedArray} structs with the highest +instruction-set-architecture extension, with each lane responsible for a separate +cell (vectoriziation over elements). In release 9.2, all matrix-free classes +have been extended with a new optional template argument specifying the +\texttt{VectorizedArrayType}. This allows users to select the vector length/ISA and, +as a consequence, the number of cells to be processed at once directly in their applications: +The deal.II-based +library \texttt{hyper.deal}~\cite{munch2020hyperdeal}, which solves the 6D Vlasov--Poisson equation with high-order +discontinuous Galerkin methods (with more than 1024 degrees of freedom per cell), works with \texttt{MatrixFree} objects of different SIMD-vector +length in the same application and benefits---in terms of performance---by the possibility of decreasing the number of cells processed by a single SIMD instruction. + +\begin{table} +\caption{Supported vector lengths of the class \texttt{VectorizedArray} and +the corresponding instruction-set-architecture extensions. }\label{tab:vectorizedarray} +\centering +\begin{tabular}{ccc} +\toprule +\textbf{double} & \textbf{float} & \textbf{ISA}\\ +\midrule +VectorizedArray & VectorizedArray & (auto-vectorization) \\ +VectorizedArray & VectorizedArray & SSE2 \\ +VectorizedArray & VectorizedArray & AVX/AVX2 \\ +VectorizedArray & VectorizedArray & AVX-512 \\ +\bottomrule +\end{tabular} + +\caption{Comparison of relevant SIMD-related classes in deal.II and C++20.}\label{tab:simd} +\centering +\begin{tabular}{cc} +\toprule +\textbf{VectorizedArray (deal.II)} & \textbf{std::simd (C++20)} \\ +\midrule +VectorizedArray & std::experimental::native\_simd \\ +VectorizedArray & std::experimental::fixed\_size\_simd \\ \bottomrule +\end{tabular} +\end{table} + +A side effect of introducing the new template argument \texttt{VectorizedArrayType} +in the \texttt{MatrixFree} classes is that any data structures +\texttt{VectorizedArrayType} can be processed if they support required +functionalities like \texttt{size()} or \texttt{value\_type}. In this context, we +would like to highlight that the new \texttt{C++20} feature \texttt{std::simd} +can be processed by the matrix-free infrastructure with minor internal +adjustment as an open pull request shows +(see \url{https://github.com/dealii/dealii/pull/9994}). +Table~\ref{tab:simd} gives a comparison of the deal.II-specific SIMD classes and +the equivalent C++20 classes. +%We welcome the standardization of the SIMD +%parallelization paradigm in C++ and intend to replace step by step our own +%wrapper class, which has been continuously developed over the last decade. We +%would like to emphasize that the work invested in this class was not in vain, +%since many performance-relevant utility functions implemented with \texttt{VectorizedArray} in mind (e.g., \texttt{vectorized\_load\_and\_transpose} +%and \texttt{vectorized\_transpose\_and\_store}) will be still used, since they +%have not become part of the standard. + +%Further additions to the \texttt{MatrixFree} infrastructure consist of: +%\begin{itemize} +%\item a new variant of \texttt{MatrixFree::cell\_loop()}: It takes two +%\texttt{std::function} objects with ranges on the locally owned degrees of freedom, one +%with work to be scheduled before the cell operation first touches some +%unknowns and another with work to be executed after the cell operation last +%touches them. The goal of +%these functions is to bring vector operations close to the time when the +%vector entries are accessed by the cell operation, which increases the cache +%hit rate of modern processors by improved temporal locality. +%\item a new form of loop \texttt{MatrixFree::loop\_cell\_centric()}: This +%kind of loop can be used in the context of discontinuous Galerkin methods, +%where both cell and face integrals have to be evaluated. While in the case of +%the traditional \texttt{loop}, cell and face integrals have been performed +%independently, the new loop performs all cell and face integrals of a cell in +%one go. This includes that each face integral has to be evaluated twice, but +%entries have to be written into the solution vector only once with improved +%data locality. Previous publications based on \texttt{deal.II} have shown the +%relevance of the latter aspect for reaching higher performance. +%\end{itemize} + +In a next step, we intend to support that users could set +\texttt{VectorizedArrayType} to \texttt{Number}, which would lead to the usage not +of \texttt{VectorizedArray} but of a specialized code-path exploiting +vectorization within an element~\cite{KronbichlerKormann2019}. + +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% +\subsection{Advances in GPU support} +\label{subsec:gpu} + +\begin{itemize} +\item overlapping of computation and communication in the case of CUDA-aware MPI +\end{itemize} + + %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% \subsection{New and improved tutorial and code gallery programs} \label{subsec:steps} - Many of the \dealii{} tutorial programs were substantially revised as - part of this release. In particular, we have converted many places - that now allow for simpler code through the use of C++11 features such - as range-based for loops and lambda functions. - - In addition, there are seven new tutorial programs: + Many of the \dealii{} tutorial programs were revised in a variety of + ways as part of this release. A particular example is that we have + converted a number of programs to use range-based for loops (a C++11 + feature) for loops over a range of integer indices such as loops over + all quadrature points or all indices of degrees of freedom during + assembly. This makes sense given that the + range-based way of writing loops seems to be the idiomatic approach + these days, and that we had previously already converted loops over + all cells in this way. + + In addition, there are a number of new tutorial programs: \begin{itemize} \item \texttt{step-47} + \todo[inline]{Zhuoran to write} \item \texttt{step-50} + \todo[inline]{Timo/Conrad/... to write} \item \texttt{step-58} + \todo[inline]{Wolfgang to write} -\item \texttt{step-65} -\todo[inline]{Martin to write} -\item \texttt{step-67} -\todo[inline]{Martin to write} +\item \texttt{step-65} presents \texttt{TransfiniteInterpolationManifold}, a +manifold class that can propagate curved boundary information into the +interior of a computational domain, and \texttt{MappingQCache}, which can sample +the information of expensive manifolds in the points of a \texttt{MappingQ} and +cache it for further use. +\item \texttt{step-67} presents an explicit time integrator for the +compressible Euler equations discretized with a high-order discontinuous +Galerkin scheme using the matrix-free infrastructure. Besides the use of +matrix-free evaluators for systems of equations and over-integration, it also +presents \texttt{MatrixFreeOperators::CellwiseInverseMassMatrix}, a fast implementation +of the action of the inverse mass matrix in the DG setting using tensor +products. Furthermore, this tutorial demonstrates the usage of new +pre and post operations, which can be passed to \texttt{cell\_loop()}, to schedule operations on sections of vectors close +to the matrix-vector product to increase data locality +and discusses performance-related aspects. \item \texttt{step-69} + \todo[inline]{Matthias/Ignacio to write} \item \texttt{step-70} + \todo[inline]{Also need to update announce and announce-short if this + makes it into the release.} \end{itemize} + \todo[inline]{Do we have new code gallery programs} + + %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% + \subsection{Support for large, fully distributed meshes} + \label{subsec:pfT} + + \todo[inline]{Peter: Write something about p::f::T} + + + + %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% + \subsection{Better support for parallel $hp$-adaptive algorithms} + \label{subsec:hp} + + \todo[inline]{Marc: Your section} + + + %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% + \subsection{Support for particle-based methods} + \label{subsec:particles} + + \todo[inline]{Luca: Your section} + + + %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% + \subsection{Python interfaces} + \label{subsec:python} + + \todo[inline]{What's new here? mention step-49 and step-53 versions written in python.} + + + %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% \subsection{Incompatible changes}