Each DA object defines the layout of two vectors: a distributed
global vector and a local vector that includes room for the
appropriate ghost points. The DA object provides information
about the size and layout of these vectors, but does not internally
allocate any associated storage space for field values. Instead, the
user can create vector objects that use the DA layout
information with the routines
ierr = DACreateGlobalVector(DA da,Vec *g); ierr = DACreateLocalVector(DA da,Vec *l);These vectors will generally serve as the building blocks for local and global PDE solutions, etc. If additional vectors with such layout information are needed in a code, they can be obtained by duplicating l or g via VecDuplicate() or VecDuplicateVecs().
We emphasize that a distributed array provides the information needed to communicate the ghost value information between processes. In most cases, several different vectors can share the same communication information (or, in other words, can share a given DA). The design of the DA object makes this easy, as each DA operation may operate on vectors of the appropriate size, as obtained via DACreateLocalVector() and DACreateGlobalVector() or as produced by VecDuplicate(). As such, the DA scatter/gather operations (e.g., DAGlobalToLocalBegin()) require vector input/output arguments, as discussed below.
PETSc currently provides no container for multiple arrays sharing the same distributed array communication; note, however, that the dof parameter handles many cases of interest.
At certain stages of many applications, there is a need to work
on a local portion of the vector, including the ghost points.
This may be done by scattering a global vector into its
local parts by using the two-stage commands
ierr = DAGlobalToLocalBegin(DA da,Vec g,InsertMode iora,Vec l); ierr = DAGlobalToLocalEnd(DA da,Vec g,InsertMode iora,Vec l);which allow the overlap of communication and computation. Since the global and local vectors, given by g and l, respectively, must be compatible with the distributed array, da, they should be generated by DACreateGlobalVector() and DACreateLocalVector() (or be duplicates of such a vector obtained via VecDuplicate()). The InsertMode can be either ADD_VALUES or INSERT_VALUES.
One can scatter the local patches into the distributed vector
with the command
ierr = DALocalToGlobal(DA da,Vec l,InsertMode mode,Vec g);Note that this function is not subdivided into beginning and ending phases, since it is purely local.
A third type of distributed array scatter is from a local
vector (including ghost points that contain irrelevant values) to
a local vector with correct ghost point values.
This scatter may be done by
commands
ierr = DALocalToLocalBegin(DA da,Vec l1,InsertMode iora,Vec l2); ierr = DALocalToLocalEnd(DA da,Vec l1,InsertMode iora,Vec l2);Since both local vectors, l1 and l2, must be compatible with the distributed array, da, they should be generated by DACreateLocalVector() (or be duplicates of such vectors obtained via VecDuplicate()). The InsertMode can be either ADD_VALUES or INSERT_VALUES.
It is possible to directly access the vector scatter contexts (see below)
used in the local-to-global ( ltog), global-to-local
( gtol), and local-to-local ( ltol)
scatters with the command
ierr = DAGetScatter(DA da,VecScatter *ltog,VecScatter *gtol,VecScatter *ltol);Most users should not need to use these contexts.