Zoltan User's Guide  |  Next  |  Previous
Note: See also hypergraph partitioning.

Graph partitioning: ParMETIS

ParMETIS is a parallel library for graph partitioning (for static load balancing) and repartitioning (for dynamic load balancing) developed at the University of Minnesota by Karypis, Schloegel and Kumar [Karypis and Kumar]. ParMETIS is therefore strictly speaking not a method but rather a collection of methods. In the Zoltan context, ParMETIS is a method with many sub-methods. Zoltan provides an interface to all the ParMETIS (sub-)methods.  The user selects which ParMETIS method to use through the parameter PARMETIS_METHOD. Most of the ParMETIS methods are based on either multilevel Kernighan-Lin partitioning or a diffusion algorithm. The names of the ParMETIS methods used by Zoltan are identical to those in the ParMETIS library. For further information about the various ParMETIS methods and parameters, please consult the ParMETIS User's Guide.

Graph partitioning is a useful abstraction for load balancing. The main idea is to represent the computational application as a weighted graph. The nodes or vertices in the graph correspond to objects in Zoltan.  Each object may have a weight that normally represents the amount of computation. The edges or arcs in the graph usually correspond to communication costs. In graph partitioning, the problem is to find a partitioning of the graph (that is,  each vertex is assigned to one out of k possible sets called partitions) that minimizes the cut size (weight) subject to the partitions having approximately equal size (weight). In repartitioning, it is assumed that a partitioning already exists. The problem is to find a good partitioning that is also "similar" in some sense to the existing partitioning. This keeps the migration cost low. All the problems described above are NP-hard so no efficient exact algorithm is known. We remark that in Zoltan 1.*, the number of partitions is always the same as the number of MPI processes (which is normally equal to the number of processors).

We give only a brief summary of the various ParMETIS methods here; for more details see the ParMETIS documentation. The methods fall into three categories:

  1. Part* - Perform graph partitioning without consideration of the initial distribution.
  2. AdaptiveRepart (ParMETIS 3) and Repart* (ParMETIS 2)  - Incremental algorithms with small migration cost.
  3. Refine* - Refines a given partitioning (balance).  Can be applied multiple times to reduce the communication cost (cut weight) if desired.
As a rule of thumb, use one of the Part* methods if you have a poor initial balance and you are willing to spend some time doing migration. One such case is static load balancing; that is, you need to balance only once. Use AdaptiveRepart or the Repart* methods when you have a reasonably good load balance that you wish to update incrementally. These methods are well suited  for dynamic load balancing (for example,  adaptive mesh refinement). A reasonable strategy is to call PartKway once to obtain a good initial balance and  later update this balance using AdaptiveRepart (Repart* in ParMetis 2.0).

Zoltan is currently compatible with ParMETIS versions 3.1 and 2.0.  There is no guarantee that Zoltan will work correctly if you have a different version of ParMETIS on your computer. (ParMETIS 3.0 will work with Zoltan in most cases, but is not officially supported. ParMETIS 3.1 is highly recommended. The 2.0 version will soon become obsolete and may not be supported in future Zoltan versions.) The ParMETIS source code can be obtained from the ParMETIS home page. As a courtesy service,  a recent, compatible version of the ParMETIS source code is distributed with Zoltan. However, ParMETIS is a completely separate library. If you do not wish to install ParMETIS, it is possible to compile Zoltan without any references to ParMETIS  (when you 'make' Zoltan, comment out the PARMETIS_LIBPATH variable in the configuration file Utilities/Config/Config.<platform>).

Note that Zoltan ignores the imbalance tolerance parameter IMBALANCE_TOL when ParMETIS 2.0 is used (the default value 1.05 is used), while IMBALANCE_TOL works correctly with ParMETIS 3.0.  Zoltan supports the multiconstraint feature of ParMETIS 3 through multiple object weights (see OBJ_WEIGHT_DIM).

The graph given to Zoltan/ParMETIS must be symmetric. Any self edges (loops) will be ignored. Multiple edges between a pair of vertices is not allowed. All weights must be non-negative. The graph does not have to be connected.
 
Method String: GRAPH or PARMETIS
Parameters:
    PARMETIS_METHOD The ParMETIS method to be used; currently nine are available. 
PartKway - multilevel Kernighan-Lin partitioning 
PartGeom - space filling curves (coordinate based) 
PartGeomKway - hybrid method based on PartKway and PartGeom (needs both  graph data and coordinates) 
AdaptiveRepart - adaptive repartitioning (only in ParMETIS 3.0 and higher)
RepartLDiffusion - diffusion algorithm (local) 
RepartGDiffusion - diffusion algorithm (global) 
RepartRemap - multilevel partioning with remap seeking to minimize migration cost 
RepartMLRemap - similar to RepartRemap but with additional multilevel refinement 
RefineKway - refine the current partitioning (balance)
The method names are case insensitive.
    PARMETIS_OUTPUT_LEVEL Amount of output the load-balancing algorithm should produce. 
0 = no output, 1 = print timing info. Turning on more bits displays more information (for example, 3=1+2, 5=1+4, 7=1+2+4).
    PARMETIS_COARSE_ALG Coarse algorithm for PartKway. 1 = serial, 2 = parallel. (ParMETIS 2 only)
    PARMETIS_SEED Random seed for ParMETIS.
    PARMETIS_ITR Ratio of interprocessor communication time to redistribution time. A high value will emphasize reducing the edge cut, while a small value will try to keep the change in the new partition (distribution) small. This parameter is only used by AdaptiveRepart. A value of between 100 and 1000 is good for most problems.
    USE_OBJ_SIZE Use (or not use) the available information about object sizes to estimate migration cost. This parameter is currently only relevant for AdaptiveRepart.
    CHECK_GRAPH Level of error checking for graph input: 0 = no checking, 1 = on-processor checking, 2 = full checking. (CHECK_GRAPH==2 is very slow and should be used only during debugging).
    SCATTER_GRAPH Scatter graph data by distributing contiguous chunks of objects (vertices) of roughly equal size to each processor before calling the partitioner. 0 = don't scatter; 1 = scatter only if all objects are on a single processor; 2 = scatter if at least one processor owns no objects (recommended to avoid a bug in ParMETIS 2.0); 3 = always scatter. 
Default values:
PARMETIS_METHOD = RepartGDiffusion
PARMETIS_OUTPUT_LEVEL = 0
PARMETIS_COARSE_ALG = 2
PARMETIS_SEED = 15
PARMETIS_ITR = 100
USE_OBJ_SIZE = 1
CHECK_GRAPH = 1
SCATTER_GRAPH = 1
Required Query Functions:
For all submethods: ZOLTAN_NUM_OBJ_FN
ZOLTAN_OBJ_LIST_FN or ZOLTAN_FIRST_OBJ_FN/ZOLTAN_NEXT_OBJ_FN pair
Only PartGeom & PartGeomKway: ZOLTAN_NUM_GEOM_FN
ZOLTAN_GEOM_MULTI_FN or ZOLTAN_GEOM_FN
All but PartGeom: ZOLTAN_NUM_EDGES_MULTI_FN or ZOLTAN_NUM_EDGES_FN
ZOLTAN_EDGE_LIST_MULTI_FN or ZOLTAN_EDGE_LIST_FN


[Table of Contents  | Next:  Jostle  |  Previous:  Refinement Tree Partitioning]