4.4.1. Creating Distributed Arrays

Up: Contents Next: Local/Global Vectors and Scatters Previous: Structured Grids Using Distributed Arrays

The PETSc DA object manages the parallel communication required while working with data stored in regular arrays. The actual data is stored in approriately sized vector objects; the DA object only contains the parallel data layout information and communication information.

One creates a distributed array communication data structure in two dimensions with the command

   ierr = DACreate2d(MPI_Comm comm,DAPeriodicType wrap,DAStencilType st,int M, 
                     int N,int m,int n,int dof,int s,int *lx,int *ly,DA *da); 
The arguments M and N indicate the global numbers of grid points in each direction, while m and n denote the processor partition in each direction; m*n must equal the number of processors in the MPI communicator, comm. Instead of specifying the processor layout, one may use PETSC_DECIDE for m and n so that PETSc will determine the partition using MPI. The type of periodicity of the array is specified by wrap, which can be DA_NONPERIODIC (no periodicity), DA_XYPERIODIC (periodic in both x- and y-directions), DA_XPERIODIC , or DA_YPERIODIC. The argument dof indicates the number of degrees of freedom at each array point, and s is the stencil width (i.e., the width of the ghost point region). The optional arrays lx and ly may contain the number of nodes along the x and y axis for each cell, i.e. the dimension of lx is m and the dimension of ly is n; or PETSC_NULL may be passed in.

Two types of distributed array communication data structures can be created, as specified by st. Star-type stencils that radiate outward only in the coordinate directions are indicated by DA_STENCIL_STAR, while box-type stencils are specified by DA_STENCIL_BOX. For example, for the two-dimensional case, DA_STENCIL_STAR with width 1 corresponds to the standard 5-point stencil, while DA_STENCIL_BOX with width 1 denotes the standard 9-point stencil. In both instances the ghost points are identical, the only difference being that with star-type stencils certain ghost points are ignored, potentially decreasing substantially the number of messages sent. Note that the DA_STENCIL_STAR stencils can save interprocessor communication in two and three dimensions.

These DA stencils have nothing directly to do with any finite difference stencils one might chose to use for a discretization; they only ensure that the correct values are in place for application of a user-defined finite difference stencil (or any other discretization technique).

The commands for creating distributed array communication data structures in one and three dimensions are analogous:

   ierr = DACreate1d(MPI_Comm comm,DAPeriodicType wrap,int M,int w,int s,int *lc,DA *inra); 
   ierr = DACreate3d(MPI_Comm comm,DAPeriodicType wrap,DAStencilType stencil_type, 
                     int M,int N,int P,int m,int n,int p,int w,int s,int *lx, 
                     int *ly,int *lz,DA *inra); 
DA_ZPERIODIC, DA_XZPERIODIC, DA_YZPERIODIC, and DA_XYZPERIODIC are additional options in three dimensions for DAPeriodicType. The routines to create distributed arrays are collective, so that all processors in the communicator comm must call DACreateXXX().


Up: Contents Next: Local/Global Vectors and Scatters Previous: Structured Grids Using Distributed Arrays