4.5.2. Scatters and Gathers

Up: Contents Next: Scattering Ghost Values Previous: Index Sets

PETSc vectors have full support for general scatters and gathers. One can select any subset of the components of a vector to insert or add to any subset of the components of another vector. We refer to these operations as generalized scatters, though they are actually a combination of scatters and gathers.

To copy selected components from one vector to another, one uses the following set of commands:

   ierr = VecScatterCreate(Vec x,IS ix,Vec y,IS iy,VecScatter *ctx); 
   ierr = VecScatterBegin(Vec x,Vec y,INSERT_VALUES,SCATTER_FORWARD,VecScatter ctx); 
   ierr = VecScatterEnd(Vec x,Vec y,INSERT_VALUES,SCATTER_FORWARD,VecScatter ctx); 
   ierr = VecScatterDestroy(VecScatter ctx); 
Here ix denotes the index set of the first vector, while iy indicates the index set of the destination vector. The vectors can be parallel or sequential. The only requirements are that the number of entries in the index set of the first vector, ix, equal the number in the destination index set, iy, and that the vectors be long enough to contain all the indices referred to in the index sets. The argument INSERT_VALUES specifies that the vector elements will be inserted into the specified locations of the destination vector, overwriting any existing values. To add the components, rather than insert them, the user should select the option ADD_VALUES instead of INSERT_VALUES.

To perform a conventional gather operation, the user simply makes the destination index set, iy, be a stride index set with a stride of one. Similarly, a conventional scatter can be done with an initial (sending) index set consisting of a stride. For parallel vectors, all processors that own the vector must call the scatter routines. When scattering from a parallel vector to sequential vectors, each processor has its own sequential vector that receives values from locations as indicated in its own index set. Similarly, in scattering from sequential vectors to a parallel vector, each processor has its own sequential vector that makes contributions to the parallel vector.

Caution: When INSERT_VALUES is used, if two different processors contribute different values to the same component in a parallel vector, either value may end up being inserted. When ADD_VALUES is used, the correct sum is added to the correct location.

In some cases one may wish to ``undo'' a scatter, that is perform the scatter backwards switching the roles of the sender and receiver. This is done by using

   ierr = VecScatterBegin(Vec y,Vec x,INSERT_VALUES,SCATTER_REVERSE,VecScatter ctx); 
   ierr = VecScatterEnd(Vec y,Vec x,INSERT_VALUES,SCATTER_REVERSE,VecScatter ctx); 
Note that the roles of the first two arguments to these routines must be swapped whenever the SCATTER_REVERSE option is used.

Once a VecScatter object has been created it may be used with any vectors that have the appropriate parallel data layout. That is, one can call VecScatterBegin() and VecScatterEnd() with different vectors than used in the call to VecScatterCreate() so long as they have the same parallel layout (number of elements on each processor are the same). Usually, these ``different'' vectors would ahve been obtained vai calls to VecDuplicate() from the original vectors used in the call to VecScatterCreate().

There is no PETSc routine that is the opposite of VecSetValues() , that is, VecGetValues(). Instead, the user should create a new vector where the components are to be stored and perform the appropriate vector scatter. For example, if one desires to obtain the values of the 100th and 200th entries of a parallel vector, p, one could use a code such as that within Figure 10 . In this example, the values of the 100th and 200th components are placed in the array values. In this example each processor now has the 100th and 200th component, but obviously each processor could gather any elements it needed, or none by creating an index set with no entries.


   Vec        p, x;         /* initial vector, destination vector */ 
   VecScatter scatter;      /* scatter context */ 
   IS         from, to;     /* index sets that define the scatter */ 
   Scalar     *values; 
   int        idx_from[] = {100,200}, idx_to[] = {0,1}; 

VecCreateSeq(PETSC_COMM_SELF,2,&x); ISCreateGeneral(PETSC_COMM_SELF,2,idx_from,&from); ISCreateGeneral(PETSC_COMM_SELF,2,idx_to,&to); VecScatterCreate(p,from,x,to,&scatter); VecScatterBegin(p,x,INSERT_VALUES,SCATTER_FORWARD,scatter); VecScatterEnd(p,x,INSERT_VALUES,SCATTER_FORWARD,scatter); VecGetArray(x,&values); ISDestroy(from); ISDestroy(to); VecScatterDestroy(scatter);


Figure 10: Example Code for Vector Scatters

The scatter comprises two stages, in order to allow overlap of communication and computation. The introduction of the VecScatter context allows the communication patterns for the scatter to be computed once and then reused repeatedly. Generally, even setting up the communication for a scatter requires communication; hence, it is best to reuse such information when possible.


Up: Contents Next: Scattering Ghost Values Previous: Index Sets