* information is provided by the
* DoFTools::extract_locally_relevant_dofs() function.
*
- * <h5>Vectors with Ghost-elements</h5>
- *
+ *
+ * <h5>Vectors with ghost elements</h5>
+ *
* A typical parallel application is dealing with two different kinds
* of parallel vectors: vectors with ghost elements (also called
- * ghosted vectors) and vectors without ghost elements. Of course
- * these might be different flavours (BlockVector, Vector; using
- * Trilinos or PETSc, etc.).
- *
- * In vectors without ghost elements knowledge about a single entry i
- * in the vector is only known to a single processor. They are
- * constructed with an IndexSet reflecting the
- * locally_owned_dofs(). There is no overlap in the IndexSets.
- * Ghosted vectors are typically created using locally_active or
- * locally_relevant IndexSets and contain elements on processors that
- * are owned by a different processor.
- *
- * One important aspect is that we forbid any modification of ghosted
- * vectors. This is because it would create subtle bugs if elements
- * are edited on one processor but do not immediately transfer to the
- * ghosted entries on the other processors.
- *
- * The usage is typically split up in the following way: ghosted
- * vectors are used for data output, postprocessing, error estimation,
- * input in integration. Vectors without ghost entries are used in all
+ * ghosted vectors) and vectors without ghost elements. (Both
+ * kinds can typically be represented by the same data type, but there
+ * are of course different vector types that can each represent both flavors:
+ * for example TrilinosWrappers::MPI::Vector, PETScWrappers::Vector, and
+ * BlockVector objects built on these).
+ * You can find a discussion of what distinguishes these kinds of vectors
+ * in the @ref GlossGhostedVector "glossary entry on ghosted vectors".
+ *
+ * From a usage point of view, ghosted vectors are typically used for
+ * data output, postprocessing, error estimation, input in
+ * integration. This is because in these operations, one typically
+ * needs access not only to @ref GlossLocallyOwnedDofs "locally owned dofs"
+ * but also to @ref GlossLocallyActiveDofs "locally active dofs"
+ * and sometimes to @ref GlossLocallyRelevantDofs "locally relevant dofs",
+ * and their values may not be stored in non-ghosted vectors on the
+ * processor that needs them. The operations listed above also only
+ * require read-only access to vectors, and ghosted vectors are therefore
+ * usable in these contexts.
+ *
+ * On the other hand, vectors without ghost entries are used in all
* other places like assembling, solving, or any other form of
- * manipulation. You can copy between vectors with and without ghost
+ * manipulation. These are typically write-only operations and
+ * therefore need not have read access to vector elements that may be
+ * owned by another processor.
+ *
+ * You can copy between vectors with and without ghost
* elements (you can see this in step-40 and step-32) using operator=.
*
+ *
* <h5>Sparsity patterns</h5>
*
* At the time of writing this, the only class equipped to deal with the
namespace parallel
{
- /**
- * A namespace for class and
- * functions that support %parallel
- * computing on %distributed memory
- * machines. See the @ref
- * distributed module for an
- * overview of the facilities this
- * namespace offers.
- *
- * @ingroup distributed
- */
+ /**
+ * A namespace for class and
+ * functions that support %parallel
+ * computing on %distributed memory
+ * machines. See the @ref
+ * distributed module for an
+ * overview of the facilities this
+ * namespace offers.
+ *
+ * @ingroup distributed
+ */
namespace distributed
{
}
* This argument is required for vectors and matrices starting with the 7.3
* release.
*
- * In olde releases we also proposed fake add/set operations. Those were the
- * cause of many bugs and deadlocks, so the usage of VectorOperation is now
- * required.
- *
* In short, you need to call compress() in the following cases (and only in
* those cases, though calling compress() in other cases just costs some
* performance):
* calls to compress().
* </dd>
*
+ * @note Compressing is an operation that only applies to vectors whose
+ * elements are uniquely owned by one and only one processor in a parallel
+ * MPI universe. It does not apply to
+ * @ref GlossGhostedVector "vectors with ghost elements".
+ *
*
* <dt class="glossary">@anchor GlossDoF <b>Degree of freedom</b></dt>
*
* dealii::Triangulation class. </dd>
*
*
+ * <dt class="glossary">@anchor GlossGhostedVector <b>Ghosted vectors</b></dt>
+ * <dd>
+ * In parallel computations, vectors come in two general kinds:
+ * without and with ghost elements. Vectors without ghost
+ * elements uniquely partition the vector elements between
+ * processors: each vector entry has exactly one processor that
+ * owns it, and this is the only processor that stores the value
+ * of this entry. In other words, if processor zero stores elements
+ * 0...49 of a vector and processor one stores elements 50...99,
+ * then processor one is out of luck accessing element 42 of this
+ * vector: it is not stored here and the value can not be assessed.
+ * This will result in an assertion.
+ *
+ * On the other hand, there are many situations where one needs to
+ * know vector elements that aren't locally owned, for example to
+ * evaluate the solution on a locally owned cell (see
+ * @ref GlossLocallyOwnedCell) for which one of the degrees of freedom
+ * is at an interface to a cell that we do not own locally (which,
+ * in this case must then be a @ref GlossGhostCell "ghost cell")
+ * and for which the neighboring cell may be the owner -- in other
+ * words, the degree of freedom is not a
+ * @ref GlossLocallyOwnedDof "locally owned" but instead only a
+ * @ref GlossLocallyActiveDof "locally active DoFs". The values of such
+ * degrees of freedom are typically stored on the machine that owns the
+ * degree of freedom and, consequently, would not be accessible on the
+ * current machine.
+ *
+ * Because one often needs these values anyway, there is a second kind of
+ * vector, often called "ghosted vector". Ghosted vectors store some elements
+ * on each processor for which that processor is not the owner.
+ * For such vectors, you can read those elements that the
+ * processor you are currently on stores but you cannot write into them
+ * because to make this work would require propagating the new value to
+ * all other processors that have a copy of this value (the list of
+ * such processors may be something which the current processor does not
+ * know and has no way of finding out efficiently). Since you cannot
+ * write into ghosted vectors, the only way to initialize such a vector
+ * is by assignment from a non-ghosted vector. This implies having to
+ * import those elements we locally want to store from other processors.
+ *
+ * The way ghosted vectors are actually stored is different between the
+ * various implementations of parallel vectors. For PETSc (and the corresponding
+ * PETScWrappers::MPI::Vector class), ghosted vectors store the same
+ * elements as non-ghosted ones would, plus some additional elements
+ * that are owned by other processors. In other words, for each element
+ * there is a clear owner among all of the processors and those elements
+ * that the current processor stores but does not own (i.e., the
+ * "ghost elements") are simply mirror images of a master value somewhere
+ * else -- thus, the name "ghost". This is also the case for the
+ * parallel::distributed::Vector class.
+ *
+ * On the other hand, in Trilinos (and consequently in
+ * TrilinosWrappers::MPI::Vector), a ghosted vector is simply a view
+ * of the parallel vector where the element distributions overlap. The
+ * 'ghosted' Trilinos vector in itself has no idea of which entries
+ * are ghosted and which are locally owned. In fact, a ghosted vector
+ * may not even store all of the elements a non-ghosted vector would
+ * store on the current processor. Consequently, for Trilinos vectors,
+ * there is no notion of an 'owner' of vector elements in the way we
+ * have it in the the non-ghost case view (or in the PETSc case) and
+ * the name "ghost element" may be misleading since in this view,
+ * every element we have available locally may or may not be stored
+ * somewhere else as well, but even if it is, the local element is not
+ * a mirror value of a master location as there is no owner of each
+ * element.
+ *
+ * @note The @ref distributed documentation module provides a brief
+ * overview of where the different kinds of vectors are typically
+ * used.
+ * </dd>
+ *
* <dt class="glossary">@anchor hp_paper <b>%hp paper</b></dt>
* <dd>The "hp paper" is a paper by W. Bangerth and O. Kayser-Herold, titled
* "Data Structures and Requirements for hp Finite Element Software", that
* <dd> Every object that makes up a Triangulation (cells, faces,
* edges, etc.), is associated with a unique number (of type
* types::manifol_id) that is used to identify which manifold object
- * is responsible to generate new points when the mesh is refined.
+ * is responsible to generate new points when the mesh is refined.
*
* By default, all manifold indicators of a mesh are set to
* numbers::invalid_manifold_id. A typical piece of code that sets the
*
* @code
* for (typename Triangulation<dim>::active_cell_iterator cell =
- * triangulation.begin_active();
- * cell != triangulation.end(); ++cell)
+ * triangulation.begin_active();
+ * cell != triangulation.end(); ++cell)
* if (cell->center()[0] < 0)
* cell->set_manifold_id (42);
* @endcode
(Denis Davydov, Wolfgang Bangerth, 2014/09/28)
</li>
+ <li> New: The glossary now contains an extensive entry on ghosted and
+ non-ghosted vectors (see @ref GlossGhostedVector
+ <br>
+ (Wolfgang Bangerth, 2014/09/27)
+ </li>
+
<li> New: Made MappingQ<dim,spacedim> aware of
Manifold<dim,spacedim>. Now we can use high order mappings that
actually follow the geometry also on the interior of codimension
* communicator)</code>, and retained until the partitioning is changed
* again. This allows for efficient parallel communication of indices. In
* particular, it stores the communication pattern, rather than having
- * to compute it again for every communication.
+ * to compute it again for every communication. (For more information on
+ * ghost vectors, see also the
+ * @ref GlossGhostedVector "glossary entry on vectors with ghost elements".)
* - Besides the usual global access operator () it is also possible to
* access vector entries in the local index space with the function @p
* local_element(). Locally owned indices are placed first, [0,
* Note that the @p insert mode of @p compress() does not set the
* elements included in ghost entries but simply discards them, assuming
* that the owning processor has set them to the desired value already.
+ * (See also the @ref GlossCompress "glossary entry on compress".)
* - The <code>update_ghost_values()</code> function imports the data from
* the owning processor to the ghost indices in order to provide read
* access to the data associated with ghosts.
* called once for a given layout. Use the constructor with
* Vector<Number> argument to create additional vectors with the same
* parallel layout.
+ *
+ * @see @ref GlossGhostedVector "vectors with ghost elements"
*/
Vector (const IndexSet &local_range,
const IndexSet &ghost_indices,
* called once for a given layout. Use the @p reinit function with
* Vector<Number> argument to create additional vectors with the same
* parallel layout.
+ *
+ * @see @ref GlossGhostedVector "vectors with ghost elements"
*/
void reinit (const IndexSet &local_range,
const IndexSet &ghost_indices,
* i.e., whenever a non-zero ghost element is found, it is compared to
* the value on the owning processor and an exception is thrown if these
* elements do not agree.
- *
*/
void compress (::dealii::VectorOperation::values operation);
* inner product will always ignore ghost elements in order to avoid
* counting the ghost data more than once. To allow writing to ghost
* elements again, call zero_out_ghosts().
+ *
+ * @see @ref GlossGhostedVector "vectors with ghost elements"
*/
void update_ghost_values () const;
* read-access to ghost elements is prohibited whereas write access is
* still possible (to those entries specified as ghosts during
* initialization), not that there are no ghost elements at all.
+ *
+ * @see @ref GlossGhostedVector "vectors with ghost elements"
*/
bool has_ghost_elements() const;
// ---------------------------------------------------------------------
//
-// Copyright (C) 2004 - 2013 by the deal.II authors
+// Copyright (C) 2004 - 2014 by the deal.II authors
//
// This file is part of the deal.II library.
//
* functions). Only the functions creating a vector of specific type differ,
* and are implemented in this particular class.
*
+ *
* <h3>Parallel communication model</h3>
*
* The parallel functionality of PETSc is built on top of the Message Passing
* operations at the same time, for example by placing zero additions if
* necessary.
*
+ * @see @ref GlossGhostedVector "vectors with ghost elements"
+ *
* @ingroup PETScWrappers
* @ingroup Vectors
* @author Wolfgang Bangerth, 2004
*
* @note This operation always creates a ghosted
* vector.
+ *
+ * @see @ref GlossGhostedVector "vectors with ghost elements"
*/
explicit Vector (const MPI_Comm &communicator,
const IndexSet &local,
*
* @note This operation always creates a ghosted
* vector.
+ *
+ * @see @ref GlossGhostedVector "vectors with ghost elements"
*/
Vector (const IndexSet &local,
const IndexSet &ghost,
/**
* Constructs a new parallel PETSc
- * vector from an Indexset. This creates a non
+ * vector from an IndexSet. This creates a non
* ghosted vector.
*/
explicit Vector (const MPI_Comm &communicator,
const IndexSet &local) DEAL_II_DEPRECATED;
+
/**
* Constructs a new parallel PETSc
- * vector from an Indexset. This creates a non
+ * vector from an IndexSet. This creates a non
* ghosted vector.
*/
explicit Vector (const IndexSet &local,
* Reinit as a ghosted vector. See
* constructor with same signature
* for more details.
+ *
+ * @see @ref GlossGhostedVector "vectors with ghost elements"
*/
void reinit (const MPI_Comm &communicator,
const IndexSet &local,
const IndexSet &ghost) DEAL_II_DEPRECATED;
/**
* Reinit as a vector without ghost elements. See
- * constructor with same signature
+ * the constructor with same signature
* for more detais.
+ *
+ * @see @ref GlossGhostedVector "vectors with ghost elements"
*/
void reinit (const IndexSet &local,
const IndexSet &ghost,
*/
void reinit (const MPI_Comm &communicator,
const IndexSet &local) DEAL_II_DEPRECATED;
+
/**
* Reinit as a vector without ghost elements. See
* constructor with same signature
* for more detais.
+ *
+ * @see @ref GlossGhostedVector "vectors with ghost elements"
*/
void reinit (const IndexSet &local,
const MPI_Comm &communicator);
// ---------------------------------------------------------------------
//
-// Copyright (C) 2004 - 2013 by the deal.II authors
+// Copyright (C) 2004 - 2014 by the deal.II authors
//
// This file is part of the deal.II library.
//
/**
* Return if the vector contains ghost
* elements.
+ *
+ * @see @ref GlossGhostedVector "vectors with ghost elements"
*/
bool has_ghost_elements() const;
// ---------------------------------------------------------------------
//
-// Copyright (C) 2008 - 2013 by the deal.II authors
+// Copyright (C) 2008 - 2014 by the deal.II authors
//
// This file is part of the deal.II library.
//
/**
* Returns if this Vector contains ghost elements.
+ *
+ * @see @ref GlossGhostedVector "vectors with ghost elements"
*/
bool has_ghost_elements() const;
// ---------------------------------------------------------------------
//
-// Copyright (C) 2008 - 2013 by the deal.II authors
+// Copyright (C) 2008 - 2014 by the deal.II authors
//
// This file is part of the deal.II library.
//
* One particular case, where an MPI message may be generated
* unexpectedly is discussed below.
*
+ *
* <h3>Accessing individual elements of a vector</h3>
*
- * Trilinos does allow read access to individual elements of a vector,
- * but in the distributed case only to elements that are stored
- * locally. We implement this through calls like
- * <tt>d=vec(i)</tt>. However, if you access an element outside the
- * locally stored range, an exception is generated.
+ * Trilinos does of course allow read access to individual
+ * elements of a vector, but in the distributed case only to
+ * elements that are stored locally. We implement this through
+ * calls like <tt>d=vec(i)</tt>. However, if you access an element
+ * outside the locally stored range, an exception is generated.
*
* In contrast to read access, Trilinos (and the respective deal.II
* wrapper classes) allow to write (or add) to individual elements of
* vectors, even if they are stored on a different process. You can do
- * this writing by writing into or adding to elements using the syntax
+ * this by writing into or adding to elements using the syntax
* <tt>vec(i)=d</tt> or <tt>vec(i)+=d</tt>,
* or similar operations. There is one catch, however, that may lead to
* very confusing error messages: Trilinos requires application programs
* Parallel vectors come in two kinds: without and with ghost elements.
* Vectors without ghost elements uniquely partition the vector elements
* between processors: each vector entry has exactly one processor that
- * owns it. For such vectors, you can read those elements that you are
- * owned by the processor you are currently on, and you can write into
+ * owns it. For such vectors, you can read those elements that
+ * the processor you are currently on owns, and you can write into
* any element whether you own it or not: if you don't own it, the
* value written or added to a vector element will be shipped to the
* processor that owns this vector element the next time you call
* compress(), as described above.
*
- * What we call a 'ghosted' vector is simply a view of the
+ * What we call a 'ghosted' vector (see
+ * @ref GlossGhostedVector "vectors with ghost elements")
+ * is simply a view of the
* parallel vector where the element distributions overlap. The 'ghosted'
* Trilinos vector in itself has no idea of which entries are ghosted and
- * which are locally owned. In particular, there is no notion of
- * an 'owner' of vector selement in the way we have it in the
- * the non-ghost case view.
+ * which are locally owned. In fact, a ghosted vector
+ * may not even store all of the elements a non-ghosted vector would
+ * store on the current processor. Consequently, for Trilinos vectors,
+ * there is no notion of an 'owner' of vector elements in the way we
+ * have it in the the non-ghost case view.
*
* This explains why we do not allow writing into ghosted vectors on the
* Trilinos side: Who would be responsible for taking care of the
* subdivides elements among processors or not, the resulting vector
* may or may not have ghost elements. See the general documentation of
* this class for more information.
+ *
+ * @see @ref GlossGhostedVector "vectors with ghost elements"
*/
explicit Vector (const Epetra_Map ¶llel_partitioning);
* subdivides elements among processors or not, the resulting vector
* may or may not have ghost elements. See the general documentation of
* this class for more information.
+ *
+ * @see @ref GlossGhostedVector "vectors with ghost elements"
*/
Vector (const Epetra_Map ¶llel_partitioning,
const VectorBase &v);
* subdivides elements among processors or not, the resulting vector
* may or may not have ghost elements. See the general documentation of
* this class for more information.
+ *
+ * @see @ref GlossGhostedVector "vectors with ghost elements"
*/
template <typename number>
void reinit (const Epetra_Map ¶llel_partitioner,
* subdivides elements among processors or not, the resulting vector
* may or may not have ghost elements. See the general documentation of
* this class for more information.
+ *
+ * @see @ref GlossGhostedVector "vectors with ghost elements"
*/
void reinit (const Epetra_Map ¶llel_partitioning,
const bool fast = false);
* subdivides elements among processors or not, the resulting vector
* may or may not have ghost elements. See the general documentation of
* this class for more information.
+ *
+ * @see @ref GlossGhostedVector "vectors with ghost elements"
*/
template <typename Number>
Vector (const Epetra_Map ¶llel_partitioning,
* subdivides elements among processors or not, the resulting vector
* may or may not have ghost elements. See the general documentation of
* this class for more information.
+ *
+ * @see @ref GlossGhostedVector "vectors with ghost elements"
*/
explicit Vector (const IndexSet ¶llel_partitioning,
const MPI_Comm &communicator = MPI_COMM_WORLD);
* subdivides elements among processors or not, the resulting vector
* may or may not have ghost elements. See the general documentation of
* this class for more information.
+ *
+ * @see @ref GlossGhostedVector "vectors with ghost elements"
*/
Vector (const IndexSet &local,
const IndexSet &ghost,
* subdivides elements among processors or not, the resulting vector
* may or may not have ghost elements. See the general documentation of
* this class for more information.
+ *
+ * @see @ref GlossGhostedVector "vectors with ghost elements"
*/
Vector (const IndexSet ¶llel_partitioning,
const VectorBase &v,
* subdivides elements among processors or not, the resulting vector
* may or may not have ghost elements. See the general documentation of
* this class for more information.
+ *
+ * @see @ref GlossGhostedVector "vectors with ghost elements"
*/
template <typename Number>
Vector (const IndexSet ¶llel_partitioning,
* subdivides elements among processors or not, the resulting vector
* may or may not have ghost elements. See the general documentation of
* this class for more information.
+ *
+ * @see @ref GlossGhostedVector "vectors with ghost elements"
*/
void reinit (const IndexSet ¶llel_partitioning,
const MPI_Comm &communicator = MPI_COMM_WORLD,
* subdivides elements among processors or not, the resulting vector
* may or may not have ghost elements. See the general documentation of
* this class for more information.
+ *
+ * @see @ref GlossGhostedVector "vectors with ghost elements"
*/
void reinit (const IndexSet &locally_owned_entries,
const IndexSet &ghost_entries,
/**
* Return if the vector contains ghost elements. This answer is true if
* there are ghost elements on at least one process.
+ *
+ * @see @ref GlossGhostedVector "vectors with ghost elements"
*/
bool has_ghost_elements() const;
projects / dealii.git / commitdiff