We recommend that application developers skip this section on a first reading. It contains information about more advanced use of PETSc vectors to improve efficiency slightly. Once an application code is fully debugged and optimized these techniques can be tried to slightly decrease memory use and improve computation speed.
There are two minor drawbacks to the basic approach described above:
ierr = VecCreateGhost(MPI_Comm comm,int n,int N,int nghost,int *ghosts,Vec *vv)or
ierr = VecCreateGhostWithArray(MPI_Comm comm,int n,int N,int nghost,int *ghosts, Scalar *array,Vec *vv)Here n is the number of local vector entries, N is the number of global entries (or PETSC_NULL) and nghost is the number of ghost entries. The array ghosts is of size nghost and contains the global vector location for each local ghost location. Using VecDuplicate() or VecDuplicateVecs() on a ghosted vector will generate additional ghosted vectors.
In many ways a ghosted vector behaves just like any other MPI vector created
by VecCreateMPI(), the difference is that the ghosted vector has an additional
``local'' representation that allows one to access the ghost locations. This is done
through the call to
ierr = VecGhostGetLocalForm(Vec g,Vec *l);The vector l is a sequential representation of the parallel vector g that shares the same array space (and hence numerical values); but allows one to access the ``ghost'' values past ``the end of the'' array. Note that one access the entries in l using the local numbering of elements and ghosts, while they are accessed in g using the global numbering.
A common usage of a ghosted vector is given by
ierr = VecGhostUpdateBegin(Vec globalin,InsertMode INSERT_VALUES,ScatterMode SCATTER_FORWARD); ierr = VecGhostUpdateEnd(Vec globalin,InsertMode INSERT_VALUES,ScatterMode SCATTER_FORWARD); ierr = VecGhostGetLocalForm(Vec globalin,Vec *localin); ierr = VecGhostGetLocalForm(Vec globalout,Vec *localout); /* Do local calculations from localin to localout */ ierr = VecGhostRestoreLocalForm(Vec globalin,Vec *localin); ierr = VecGhostRestoreLocalForm(Vec globalout,Vec *localout); ierr = VecGhostUpdateBegin(Vec globalout,InsertMode ADD_VALUES,ScatterMode SCATTER_REVERSE); ierr = VecGhostUpdateEnd(Vec globalout,InsertMode ADD_VALUES,ScatterMode SCATTER_REVERSE);The routines VecGhostUpdateBegin/End() are equivalent to the routines VecScatterBegin/End() above except that since they are scattering into the ghost locations, they do not need to copy the local vector values, which are already in place. In addition, the user does not have to allocate the local work vector, since the ghosted vector already has allocated slots to contain the ghost values.
The input arguments INSERT_VALUES and SCATTER_FORWARD cause the ghost values to be correctly updated from the appropriate processor. The arguments ADD_VALUES and SCATTER_REVERSE update the ``local'' portions of the vector from all the other processors' ghost values. This would be appropriate, for example, when performing a finite element assembly of a load vector.
Section Partitioning discusses the important topic of partitioning an unstructured grid.