The PETSc DA object manages the parallel communication required while working with data stored in regular arrays. The actual data is stored in approriately sized vector objects; the DA object only contains the parallel data layout information and communication information.
One creates a distributed array communication data structure
in two dimensions with the command
ierr = DACreate2d(MPI_Comm comm,DAPeriodicType wrap,DAStencilType st,int M, int N,int m,int n,int dof,int s,int *lx,int *ly,DA *da);The arguments M and N indicate the global numbers of grid points in each direction, while m and n denote the processor partition in each direction; m*n must equal the number of processors in the MPI communicator, comm. Instead of specifying the processor layout, one may use PETSC_DECIDE for m and n so that PETSc will determine the partition using MPI. The type of periodicity of the array is specified by wrap, which can be DA_NONPERIODIC (no periodicity), DA_XYPERIODIC (periodic in both x- and y-directions), DA_XPERIODIC , or DA_YPERIODIC. The argument dof indicates the number of degrees of freedom at each array point, and s is the stencil width (i.e., the width of the ghost point region). The optional arrays lx and ly may contain the number of nodes along the x and y axis for each cell, i.e. the dimension of lx is m and the dimension of ly is n; or PETSC_NULL may be passed in.
Two types of distributed array communication data structures can be created, as specified by st. Star-type stencils that radiate outward only in the coordinate directions are indicated by DA_STENCIL_STAR, while box-type stencils are specified by DA_STENCIL_BOX. For example, for the two-dimensional case, DA_STENCIL_STAR with width 1 corresponds to the standard 5-point stencil, while DA_STENCIL_BOX with width 1 denotes the standard 9-point stencil. In both instances the ghost points are identical, the only difference being that with star-type stencils certain ghost points are ignored, potentially decreasing substantially the number of messages sent. Note that the DA_STENCIL_STAR stencils can save interprocessor communication in two and three dimensions.
These DA stencils have nothing directly to do with any finite difference stencils one might chose to use for a discretization; they only ensure that the correct values are in place for application of a user-defined finite difference stencil (or any other discretization technique).
The commands for creating distributed array communication data structures
in one and three dimensions are analogous:
ierr = DACreate1d(MPI_Comm comm,DAPeriodicType wrap,int M,int w,int s,int *lc,DA *inra); ierr = DACreate3d(MPI_Comm comm,DAPeriodicType wrap,DAStencilType stencil_type, int M,int N,int P,int m,int n,int p,int w,int s,int *lx, int *ly,int *lz,DA *inra);DA_ZPERIODIC, DA_XZPERIODIC, DA_YZPERIODIC, and DA_XYZPERIODIC are additional options in three dimensions for DAPeriodicType. The routines to create distributed arrays are collective, so that all processors in the communicator comm must call DACreateXXX().