MOAB: Mesh Oriented datABase
(version 5.2.1)
|
00001 /* 00002 * ParCommGraph.hpp 00003 * 00004 * will be used to setup communication between 2 distributed meshes, in which one mesh was migrated 00005 * from the other. (one example is atmosphere mesh migrated to coupler pes) 00006 * 00007 * there are 3 communicators in play, one for each mesh, and one for the joined 00008 * communicator, that spans both sets of processes; to send mesh or tag data we need to use the 00009 * joint communicator, use nonblocking MPI_iSend and blocking MPI_Recv receives 00010 * 00011 * various methods should be available to migrate meshes; trivial, using graph partitioner (Zoltan 00012 * PHG) and using a geometric partitioner (Zoltan RCB) 00013 * 00014 * communicators are represented by their MPI groups, not by their communicators, because 00015 * the groups are always defined, irrespective of what tasks are they on. Communicators can be 00016 * MPI_NULL, while MPI_Groups are always defined 00017 * 00018 * Some of the methods in here are executed over the sender communicator, some are over the 00019 * receiver communicator They can switch places, what was sender becomes the receiver and viceversa 00020 * 00021 * The name "graph" is in the sense of a bipartite graph, in which we can separate senders and 00022 * receivers tasks 00023 * 00024 * The info stored in the ParCommGraph helps in migrating fields (MOAB tags) from component to the 00025 * coupler and back 00026 * 00027 * So initially the ParCommGraph is assisting in mesh migration (from component to coupler) and 00028 * then is used to migrate tag data from component to coupler and back from coupler to component. 00029 * 00030 * The same class is used after intersection (which is done on the coupler pes between 2 different 00031 * component migrated meshes) and it alters communication pattern between the original component pes 00032 * and coupler pes; 00033 * 00034 * We added a new way to send tags between 2 models; the first application of the new method is to 00035 * send tag from atm dynamics model (spectral elements, with np x np tags defined on each element, 00036 * according to the GLOBAL_DOFS tag associated to each element) towards the atm physics model, which 00037 * is just a point cloud of vertices distributed differently to the physics model pes; matching is 00038 * done using GLOBAL_ID tag on vertices; Right now, we assume that the models are on different pes, 00039 * but the joint communicator covers both and that the ids of the tasks are with respect to the 00040 * joint communicator 00041 * 00042 * 00043 */ 00044 #include "moab_mpi.h" 00045 #include "moab/Interface.hpp" 00046 #include "moab/ParallelComm.hpp" 00047 #include <map> 00048 00049 #ifndef SRC_PARALLEL_MOAB_PARCOMMGRAPH_HPP_ 00050 #define SRC_PARALLEL_MOAB_PARCOMMGRAPH_HPP_ 00051 00052 namespace moab 00053 { 00054 00055 class ParCommGraph 00056 { 00057 public: 00058 enum TypeGraph 00059 { 00060 INITIAL_MIGRATE, 00061 COVERAGE, 00062 DOF_BASED 00063 }; 00064 virtual ~ParCommGraph(); 00065 00066 /** 00067 * \brief collective constructor, will be called on all sender tasks and receiver tasks 00068 * \param[in] joincomm joint MPI communicator that covers both sender and receiver MPI groups 00069 * \param[in] group1 MPI group formed with sender tasks; (sender usually loads the mesh in a 00070 * migrate scenario) \param[in] group2 MPI group formed with receiver tasks; (receiver 00071 * usually receives the mesh in a migrate scenario) \param[in] coid1 sender component unique 00072 * identifier in a coupled application (in climate simulations could be the Atmosphere Comp id = 00073 * 5 or Ocean Comp ID , 17) \param[in] coid2 receiver component unique identifier in a 00074 * coupled application (it is usually the coupler, 2, in E3SM) 00075 * 00076 * this graph will be formed on sender and receiver tasks, and, in principle, will hold info 00077 * about how the local entities are distributed on the other side 00078 * 00079 * Its major role is to help in migration of data, from component to the coupler and vice-versa; 00080 * Each local entity has a corresponding task (or tasks) on the other side, to where the data 00081 * needs to be sent 00082 * 00083 * important data stored in ParCommGraph, immediately it is created 00084 * - all sender and receiver tasks ids, with respect to the joint communicator 00085 * - local rank in sender and receiver group (-1 if not part of the respective group) 00086 * - rank in the joint communicator (from 0) 00087 */ 00088 ParCommGraph( MPI_Comm joincomm, MPI_Group group1, MPI_Group group2, int coid1, int coid2 ); 00089 00090 /** 00091 * \brief copy constructor will copy only the senders, receivers, compid1, etc 00092 */ 00093 ParCommGraph( const ParCommGraph& ); 00094 00095 /** 00096 \brief Based on the number of elements on each task in group 1, partition for group 2, 00097 trivially 00098 00099 <B>Operations:</B> it is called on every receiver task; decides how are all elements distributed 00100 00101 Note: establish how many elements are sent from each task in group 1 to tasks in group 2 00102 This call is usually made on a root / master process, and will construct local maps that 00103 are member data, which contain the communication graph, in both directions Also, number of 00104 elements migrated/exchanged between each sender/receiver 00105 00106 \param[in] numElemsPerTaskInGroup1 (std::vector<int> &) number of elements on each sender 00107 task 00108 */ 00109 00110 ErrorCode compute_trivial_partition( std::vector< int >& numElemsPerTaskInGroup1 ); 00111 00112 /** 00113 \brief pack information about receivers view of the graph, for future sending to receiver 00114 root 00115 00116 <B>Operations:</B> Local, called on root process of the senders group 00117 00118 \param[out] packed_recv_array 00119 packed data will be sent to the root of receivers, and distributed from there, and 00120 will have this information, for each receiver, concatenated 00121 receiver 1 task, number of senders for receiver 1, then sender tasks for receiver 1, 00122 receiver 2 task, number of senders for receiver 2, sender tasks for receiver 2, etc Note: only 00123 the root of senders will compute this, and send it over to the receiver root, which will 00124 distribute it over each receiver; We do not pack the sizes of data to be sent, only the 00125 senders for each of the receivers (could be of size O(n^2) , where n is the number of tasks ; 00126 but in general, it should be more like O(n) ). Each sender sends to a "finite" number of 00127 receivers, and each receiver receives from a finite number of senders). We need this info to 00128 decide how to set up the send/receive waiting game for non-blocking communication ) 00129 */ 00130 ErrorCode pack_receivers_graph( std::vector< int >& packed_recv_array ); 00131 00132 // get methods for private data 00133 bool is_root_sender() 00134 { 00135 return rootSender; 00136 } 00137 00138 bool is_root_receiver() 00139 { 00140 return rootReceiver; 00141 } 00142 00143 int sender( int index ) 00144 { 00145 return senderTasks[index]; 00146 } 00147 00148 int receiver( int index ) 00149 { 00150 return receiverTasks[index]; 00151 } 00152 00153 int get_component_id1() 00154 { 00155 return compid1; 00156 } 00157 int get_component_id2() 00158 { 00159 return compid2; 00160 } 00161 00162 int get_context_id() 00163 { 00164 return context_id; 00165 } 00166 void set_context_id( int other_id ) 00167 { 00168 context_id = other_id; 00169 } 00170 00171 EntityHandle get_cover_set() 00172 { 00173 return cover_set; 00174 } 00175 void set_cover_set( EntityHandle cover ) 00176 { 00177 cover_set = cover; 00178 } 00179 00180 // return local graph for a specific task 00181 ErrorCode split_owned_range( int sender_rank, Range& owned ); 00182 00183 ErrorCode split_owned_range( Range& owned ); 00184 00185 ErrorCode send_graph( MPI_Comm jcomm ); 00186 00187 ErrorCode send_graph_partition( ParallelComm* pco, MPI_Comm jcomm ); 00188 00189 ErrorCode send_mesh_parts( MPI_Comm jcomm, ParallelComm* pco, Range& owned ); 00190 00191 // this is called on receiver side 00192 ErrorCode receive_comm_graph( MPI_Comm jcomm, ParallelComm* pco, std::vector< int >& pack_array ); 00193 00194 ErrorCode receive_mesh( MPI_Comm jcomm, ParallelComm* pco, EntityHandle local_set, 00195 std::vector< int >& senders_local ); 00196 00197 ErrorCode release_send_buffers(); 00198 00199 ErrorCode send_tag_values( MPI_Comm jcomm, ParallelComm* pco, Range& owned, std::vector< Tag >& tag_handles ); 00200 00201 ErrorCode receive_tag_values( MPI_Comm jcomm, ParallelComm* pco, Range& owned, std::vector< Tag >& tag_handles ); 00202 00203 // getter method 00204 const std::vector< int >& senders() 00205 { 00206 return senderTasks; 00207 } // reference copy; refers to sender tasks in joint comm 00208 const std::vector< int >& receivers() 00209 { 00210 return receiverTasks; 00211 } 00212 00213 ErrorCode settle_send_graph( TupleList& TLcovIDs ); 00214 00215 // this will set after_cov_rec_sizes 00216 void SetReceivingAfterCoverage( 00217 std::map< int, std::set< int > >& idsFromProcs ); // will make sense only on receivers, right now after cov 00218 00219 // strideComp is np x np, or 1, in our cases 00220 // will fill up ordered lists for corresponding IDs on the other component 00221 // will form back and forth information, from ordered list of IDs, to valuesComp 00222 void settle_comm_by_ids( int comp, TupleList& TLBackToComp, std::vector< int >& valuesComp ); 00223 // new partition calculation 00224 ErrorCode compute_partition( ParallelComm* pco, Range& owned, int met ); 00225 00226 // dump local information about graph 00227 ErrorCode dump_comm_information( std::string prefix, int is_send ); 00228 00229 private: 00230 /** 00231 \brief find ranks of a group with respect to an encompassing communicator 00232 00233 <B>Operations:</B> Local, usually called on root process of the group 00234 00235 \param[in] joincomm (MPI_Comm) 00236 \param[in] group (MPI_Group) 00237 \param[out] ranks ( std::vector<int>) ranks with respect to the joint communicator 00238 */ 00239 void find_group_ranks( MPI_Group group, MPI_Comm join, std::vector< int >& ranks ); 00240 00241 MPI_Comm comm; 00242 std::vector< int > senderTasks; // these are the sender tasks in joint comm 00243 std::vector< int > receiverTasks; // these are all the receiver tasks in joint comm 00244 bool rootSender; 00245 bool rootReceiver; 00246 int rankInGroup1, rankInGroup2; // group 1 is sender, 2 is receiver 00247 int rankInJoin, joinSize; 00248 int compid1, compid2; 00249 int context_id; // used to identify the other comp for intersection 00250 EntityHandle cover_set; // will be initialized only if it is the receiver parcomm graph, in 00251 // CoverageGraph 00252 00253 // communication graph from group1 to group2; 00254 // graph[task1] = vec1; // vec1 is a stl vector of tasks in group2 00255 std::map< int, std::vector< int > > recv_graph; // to what tasks from group2 to send (actual communication graph) 00256 std::map< int, std::vector< int > > 00257 recv_sizes; // how many elements to actually send from a sender task to receiver tasks 00258 std::map< int, std::vector< int > > 00259 sender_graph; // to what tasks from group2 to send (actual communication graph) 00260 std::map< int, std::vector< int > > 00261 sender_sizes; // how many elements to actually send from a sender task to receiver tasks 00262 00263 std::vector< ParallelComm::Buffer* > localSendBuffs; // this will store the pointers to the Buffers 00264 // will be released only when all mpi requests are waited 00265 // for 00266 int* comm_graph; // this will store communication graph, on sender master, sent by nonblocking 00267 // send to the master receiver first integer will be the size of the graph, 00268 // the rest will be the packed graph, for trivial partition 00269 00270 // these will be now used to store ranges to be sent from current sender to each receiver in 00271 // joint comm 00272 std::map< int, Range > split_ranges; 00273 00274 std::vector< MPI_Request > sendReqs; // there will be multiple requests, 2 for comm graph, 2 for each Buffer 00275 // there are as many buffers as sender_graph[rankInJoin].size() 00276 00277 // active on both receiver and sender sides 00278 std::vector< int > corr_tasks; // subset of the senderTasks, in the joint comm for sender; 00279 // subset of receiverTasks for receiver side 00280 std::vector< int > corr_sizes; // how many primary entities corresponding to the other side 00281 // so what we know is that the local range corresponds to remote corr_sizes[i] size ranges on 00282 // tasks corr_tasks[i] 00283 00284 // these will be used now after coverage, quick fix; they will also be populated by 00285 // iMOAB_CoverageGraph 00286 TypeGraph graph_type; // this should be false , set to true in settle send graph, to use send_IDs_map 00287 std::map< int, std::vector< int > > involved_IDs_map; // replace send and recv IDs_mapp with involved_IDs_map 00288 // used only for third method: DOF_BASED 00289 std::map< int, std::vector< int > > 00290 map_index; // from index in involved[] to index in values[] of tag, for each corr task 00291 std::map< int, std::vector< int > > map_ptr; // lmap[ie], lmap[ie+1], pointer into map_index[corrTask] 00292 }; 00293 00294 } // namespace moab 00295 #endif /* SRC_PARALLEL_MOAB_PARCOMMGRAPH_HPP_ */