From 74c8157efd3282dd11459fa28f0b92fcc7fef9c9 Mon Sep 17 00:00:00 2001
From: bangerth
- deal.II can interface to the
- PETSc and
- Trilinos software
- libraries. Both of these libraries provide lots of functions for linear
- algebra, among other things, for example implementations of a variety of
- linear solvers, as well as various different sparse and dense matrix and
- vector formats. Trilinos also has many subpackages that deal with
- problems that go far beyond linear algebra, for example nonlinear
- solvers, automatic differentiation packages, uncertainty propagation
- engines, etc. Of particular interest to deal.II is their ability to
- provide this functionality both on sequential and parallel (using MPI)
- computers. PETSc is written in C; Trilinos is written in C++ and can be
- considered to be a more modern version of PETSc though both packages are
- under continuing development at their respective national laboratories.
-
- deal.II has wrapper classes to the linear algebra
- parts of both packages that provide almost the
- same interfaces as the built-in deal.II linear
- algebra classes. We use these interfaces for parallel computations based
- on MPI since the native deal.II linear algebra classes lack this ability.
-
- The use of PETSc and Trilinos is optional. To use the wrapper classes,
- you first have to install these packages and
- point deal.II's
- PETSc usually requires you to set the
- environment variables
- Alternatively, the
- There is an additional caveat: PETSc appears not to co-operate
- well when using threads and some programs crash when deal.II is
- compiled in its usual mode supporting multithreading. If you see
- this sort of behavior, we recommend to try disabling
- multithreading upon configuration of deal.II
- using the
- Installing both PETSc and deal.II together can be a bit of a
- challenge. A good summary of the relevant steps can be found on
- the Frequently Asked Questions page.
- Note: deal.II can be installed with both
- PETSc and Trilinos (see below) and they do not usually get in their
- respective ways. There are, however, occasions where this is not true
- and this fundamentally comes from the fact that both of these packages
- are built from subpackages that are developed by independent
- groups. Unfortunately, some of these sub-packages can be configured to
- be part of both PETSc and Trilinos, and if you try to
- use deal.II with versions of PETSc and Trilinos
- that both contain a particular sub-package, little good will come
- of it in general. In particular, we have experienced this with the ML
- package that can serve as an algebraic multigrid method to both PETSc
- and Trilinos. If both of these packages are configured to use ML, then
- difficult to understand error messages at compile or link time are
- almost inevitable, and there is little the deal.II
- build system can do to prevent this. Thus, don't try to do such a
- thing!
-
- As above, set the
- deal.II and its tutorial programs use several of the
- Trilinos sub-packages.
-
- Note: Trilinos versions 10.6.x, 10.8.0, 10.8.1, 10.10.2, 10.12.1 are not
- compatible with deal.II. They contain subtle bugs related to (parallel)
- matrices and vectors. Versions tested to work are 10.4.2, 10.8.5, and
- 10.12.2. We recommend only using one of the tested versions for the time
- being.
-
- Trilinos uses cmake to configure and
- build. The following slightly longish set of commands will set up a
- reasonable configuration:
-
- Trilinos sometimes searches for other libraries but can't find
- them if they are not in the usual directories or have other
- names. A common example are BLAS or LAPACK. In a case like
- this, you may have to specifically pass the directories and/or
- library names under which they can be found
- to
- A system wide installation Support for Trilinos will be enabled
- automatically if a system wide installation of Trilinos can
- be found. If the Trilinos installation does not reside under a
- default system location provide TRILINOS_DIR in the same fashion as
- for the self compiled case:
- Interfacing deal.II to PETSc and Trilinos
-
- About PETSc, Trilinos, and the deal.II
- interfaces
-
- Configuring the interfaces to PETSc and Trilinos
-
- ./configure
to the
- installation directories. This happens in similar ways for the two
- packages:
- PETSc
-
- PETSC_DIR
and PETSC_ARCH
- to a path to PETSc and denoting the architecture for which PETSc is
- compiled. If these environment variables are set, then
- deal.II will pick them up during
- configuration, and store them. It will then also recognize that
- PETSc shall be used, and enable the wrapper classes.
- -DPETSC_DIR=DIR
and
- -DPETSC_ARCH=ARCH
options for cmake
- can be used to override the values of PETSC_DIR
- and PETSC_ARCH
or if these environment
- variables are not set at all. If you do have a PETSc
- installation and have set the PETSC_DIR
and
- PETSC_ARCH
environment variables but do not wish
- deal.II to be configured for PETSc use, you
- should specify -DDEAL_II_WITH_PETSC=OFF
as a flag
- during configuration.
- -DDEAL_II_WITH_THREADS=OFF
switch for
- cmake
.
- Trilinos
-
- TRILINOS_DIR
- environment variable to the path to an existing Trilinos installation,
- or use the -DTRILINOS_DIR=/path/to/trilinos
switch of
- the deal.II cmake
script. It should point to the path
- of which the include and lib directories are subdirs.
- Trilinos starting with version 10.0
-
-
-
- Again, the path into which you want to install Trilinos in the second to
- last line of the cmake command needs to be adjusted. Obviously, if you
- want to use Trilinos with MPI on parallel machines, you also need to
- flip the value of the
- cd trilinos-10.4.2
- mkdir build
- cd build
-
- cmake -D Trilinos_ENABLE_OPTIONAL_PACKAGES:BOOL=ON \
- -D Trilinos_ENABLE_Sacado:BOOL=ON \
- -D Trilinos_ENABLE_Stratimikos:BOOL=ON \
- -D CMAKE_BUILD_TYPE:STRING=RELEASE \
- -D CMAKE_CXX_FLAGS:STRING="-g -O3" \
- -D CMAKE_C_FLAGS:STRING="-g -O3" \
- -D CMAKE_FORTRAN_FLAGS:STRING="-g -O5" \
- -D Trilinos_EXTRA_LINK_FLAGS:STRING="-lgfortran" \
- -D CMAKE_VERBOSE_MAKEFILE:BOOL=FALSE \
- -D Trilinos_VERBOSE_CONFIGURE:BOOL=FALSE \
- -D TPL_ENABLE_MPI:BOOL=OFF \
- -D CMAKE_INSTALL_PREFIX:PATH=/w/bangerth/share/x86_64/trilinos-10.4.2 \
- -D BUILD_SHARED_LIBS:BOOL=ON ..
-
- make
- make install
-
- TPL_ENABLE_MPI
flag above. Finally,
- if your computer has more than one processor core, you may want to add
- the -jN
flag to the calls to make
- where N
is the number of compile jobs you want to run in
- parallel.
- cmake
. For example, this may mean to add the
- following flags to the call above:
-
-
-
-
- -D BLAS_LIBRARY_NAMES:STRING=goto \
- -D BLAS_LIBRARY_DIRS:STRING=/apps/GotoBLAS/lib64 \
- -D LAPACK_LIBRARY_NAMES:STRING=lapack \
- -D LAPACK_LIBRARY_DIRS:STRING=/apps/lapack-3.2.1/lib64
-
- Configuring for installed Trilinos packages
-
-
-
- -DTRILINOS_DIR=/path/to/trilinos
-
-
+ PETSc is a + software package that provides lots of functionality for linear + algebra, among other things. For example, it includes implementations of a variety of + linear solvers, as well as various different sparse and dense matrix and + vector formats. Of particular interest to deal.II is their ability to + provide this functionality both on sequential and parallel (using MPI) + computers. +
+ ++ deal.II has wrapper classes to the linear algebra + parts of PETSc that provide almost the + same interfaces as the built-in deal.II linear + algebra classes. We use these interfaces for parallel computations based + on MPI since the native deal.II linear algebra classes lack this + ability. They are used, among other programs, in step-17, step-18 and + step-40. +
+ ++ For a general overview of building deal.II with + PETSc, first see the ReadMe file. +
+ +
+ PETSc usually requires you to set the
+ environment variables PETSC_DIR
and PETSC_ARCH
+ to a path to PETSc and denoting the architecture for which PETSc is
+ compiled. If these environment variables are set, then
+ deal.II will pick them up during
+ configuration, and store them. It will then also recognize that
+ PETSc shall be used, and enable the wrapper classes.
+
+ Alternatively, the -DPETSC_DIR=DIR
and
+ -DPETSC_ARCH=ARCH
options for cmake
+ can be used to override the values of PETSC_DIR
+ and PETSC_ARCH
or if these environment
+ variables are not set at all. If you do have a PETSc
+ installation and have set the PETSC_DIR
and
+ PETSC_ARCH
environment variables but do not wish
+ deal.II to be configured for PETSc use, you
+ should specify -DDEAL_II_WITH_PETSC=OFF
as a flag
+ during configuration.
+
+ Note: PETSc appears not to co-operate
+ well when using threads and some programs crash when deal.II is
+ compiled in its usual mode supporting multithreading. If you see
+ this sort of behavior, disable
+ multithreading upon configuration of deal.II
+ using the -DDEAL_II_WITH_THREADS=OFF
switch for
+ cmake
.
+
Note: deal.II can be installed with both + PETSc and Trilinos and they do not usually get in their + respective ways. There are, however, occasions where this is not true + and this fundamentally comes from the fact that both of these packages + are built from subpackages that are developed by independent + groups. Unfortunately, some of these sub-packages can be configured to + be part of both PETSc and Trilinos, and if you try to + use deal.II with versions of PETSc and Trilinos + that both contain a particular sub-package, little good will come + of it in general. In particular, we have experienced this with the ML + package that can serve as an algebraic multigrid method to both PETSc + and Trilinos. If both of these packages are configured to use ML, then + difficult to understand error messages at compile or link time are + almost inevitable, and there is little the deal.II + build system can do to prevent this. Thus, don't try to do that! +
+ + ++ Installing PETSc correctly can be a bit of a + challenge. To start, take a look at + the PETSc installation instructions. We have found that + the following steps generally appear to work where we simply unpack and + build PETSc in its final location (i.e., we do not first build and then + install it into a separate directory): +
+ + tar xvzf petsc-x-y-z.tar.gz + cd petsc-x-y-z + export PETSC_DIR=`pwd` + export PETSC_ARCH=x86_64 # or any other identifying text for your machine + export LD_LIBRARY_PATH=$PETSC_DIR/$PETSC_ARCH/lib:$LD_LIBRARY_PATH + ./config/configure.py --with-shared=1 --with-x=0 --with-mpi=1 --download-hypre=1 + make ++ + +
+ This automatically builds PETSc with both MPI and the algebraic
+ multigrid preconditioner package Hypre (which we use in step-40). You
+ may wish to put the export
commands into
+ your ~/.bashrc
or ~/.cshrc
files, with the
+ first one replaced by something of the kind
+
+ + export PETSC_DIR=/path/to/petsc-x-y-z ++ + +
+ Trilinos is a + software package that provides lots of functionality for linear + algebra, among other things. For example, it includes implementations of a variety of + linear solvers, as well as various different sparse and dense matrix and + vector formats. Trilinos also has many subpackages that deal with + problems that go far beyond linear algebra, for example nonlinear + solvers, automatic differentiation packages, uncertainty propagation + engines, etc. Of particular interest to deal.II is their ability to + provide this functionality both on sequential and parallel (using MPI) + computers. Compared to PETSc, which is written in C, Trilinos is written in + C++ and can be + considered to be a more modern version of PETSc though both packages are + under continuing development at their respective national laboratories. +
+ ++ deal.II has wrapper classes to the linear algebra + parts of Trilinos that provide almost the + same interfaces as the built-in deal.II linear + algebra classes. We use these interfaces for parallel computations based + on MPI since the native deal.II linear algebra classes lack this + ability. They are used, among other programs, in step-31 and step-32. +
+ ++ While building deal.II with Trilinos is covered in + the ReadMe file, we here give an + introduction to building Trilinos in such a way that it contains + everything that we need from the deal.II side. +
+ + ++ Note: Trilinos versions 10.6.x, 10.8.0, 10.8.1, 10.10.2, 10.12.1 are not + compatible with deal.II. They contain subtle bugs related to (parallel) + matrices and vectors. Versions tested to work are 10.4.2, 10.8.5, and + 10.12.2. We recommend only using one of the tested versions for the time + being. +
+ ++ Trilinos uses cmake to configure and + build. The following slightly longish set of commands will set up a + reasonable configuration: +
+ + cd trilinos-10.12.2 + mkdir build + cd build + + cmake -D Trilinos_ENABLE_OPTIONAL_PACKAGES:BOOL=ON \ + -D Trilinos_ENABLE_Sacado:BOOL=ON \ + -D Trilinos_ENABLE_Stratimikos:BOOL=ON \ + -D CMAKE_BUILD_TYPE:STRING=RELEASE \ + -D CMAKE_CXX_FLAGS:STRING="-g -O3" \ + -D CMAKE_C_FLAGS:STRING="-g -O3" \ + -D CMAKE_FORTRAN_FLAGS:STRING="-g -O5" \ + -D Trilinos_EXTRA_LINK_FLAGS:STRING="-lgfortran" \ + -D CMAKE_VERBOSE_MAKEFILE:BOOL=FALSE \ + -D Trilinos_VERBOSE_CONFIGURE:BOOL=FALSE \ + -D TPL_ENABLE_MPI:BOOL=OFF \ + -D CMAKE_INSTALL_PREFIX:PATH=/w/bangerth/share/x86_64/trilinos-10.12.2 \ + -D BUILD_SHARED_LIBS:BOOL=ON .. + + make + make install ++ You will need to adjust the path into which you want to install Trilinos + in the second to last line of the cmake command. + + +
+ Using MPI: If you
+ want to use Trilinos with MPI on parallel machines,
+ use TPL_ENABLE_MPI=ON
instead.
+
+ Parallel builds:
+ If your computer has more than one processor core, use
+ make -jN
instead of make
in the last two lines
+ above, where N
is the number of processors you have.
+
+ Trilinos sometimes searches for other libraries but can't find
+ them if they are not in the usual directories or have other
+ names. A common example are BLAS or LAPACK. In a case like
+ this, you may have to specifically pass the directories and/or
+ library names under which they can be found
+ to cmake
. For example, this may mean to add the
+ following flags to the call above:
+
+ + -D BLAS_LIBRARY_NAMES:STRING=goto \ + -D BLAS_LIBRARY_DIRS:STRING=/apps/GotoBLAS/lib64 \ + -D LAPACK_LIBRARY_NAMES:STRING=lapack \ + -D LAPACK_LIBRARY_DIRS:STRING=/apps/lapack-3.2.1/lib64 ++ + +
cmake
automatically finds it, use -DDEAL_II_WITH_PETSC=OFF
More information on configuring and building PETSC can be
- found here.
+ found here.
@@ -672,7 +672,7 @@
cmake
automatically finds it, use
-DDEAL_II_WITH_TRILINOS=OFF
More details about
compatibility and configuration can be found
- here.
+ here.
--
2.39.5