ParCommGraph.hpp

Go to the documentation of this file.

/*
 * ParCommGraph.hpp
 *
 *  will be used to setup communication between 2 distributed meshes, in which one mesh was migrated
 * from the other. (one example is atmosphere mesh migrated to coupler pes)
 *
 *  there are 3 communicators in play, one for each mesh, and one for the joined
 *  communicator, that spans both sets of processes; to send mesh or tag data we need to use the
 * joint communicator, use nonblocking MPI_iSend and blocking MPI_Recv receives
 *
 *  various methods should be available to migrate meshes; trivial, using graph partitioner (Zoltan
 * PHG) and using a geometric partitioner  (Zoltan RCB)
 *
 *  communicators are represented by their MPI groups, not by their communicators, because
 *  the groups are always defined, irrespective of what tasks are they on. Communicators can be
 * MPI_NULL, while MPI_Groups are always defined
 *
 *  Some of the methods in here are executed over the sender communicator, some are over the
 * receiver communicator They can switch places, what was sender becomes the receiver and viceversa
 *
 *  The name "graph" is in the sense of a bipartite graph, in which we can separate senders and
 * receivers tasks
 *
 *  The info stored in the ParCommGraph helps in migrating fields (MOAB tags) from component to the
 * coupler and back
 *
 *  So initially the ParCommGraph is assisting in mesh migration (from component to coupler) and
 * then is used to migrate tag data from component to coupler and back from coupler to component.
 *
 *  The same class is used after intersection (which is done on the coupler pes between 2 different
 * component migrated meshes) and it alters communication pattern between the original component pes
 * and coupler pes;
 *
 *   We added a new way to send tags between 2 models; the first application of the new method is to
 * send tag from atm dynamics model (spectral elements, with np x np tags defined on each element,
 * according to the GLOBAL_DOFS tag associated to each element) towards the atm physics model, which
 * is just a point cloud of vertices distributed differently to the physics model pes; matching is
 * done using GLOBAL_ID tag on vertices; Right now, we assume that the models are on different pes,
 * but the joint communicator covers both and that the ids of the tasks are with respect to the
 * joint communicator
 *
 *
 */
#include "moab_mpi.h"
#include "moab/Interface.hpp"
#include "moab/ParallelComm.hpp"
#include 

#ifndef SRC_PARALLEL_MOAB_PARCOMMGRAPH_HPP_
#define SRC_PARALLEL_MOAB_PARCOMMGRAPH_HPP_

namespace moab
{

class ParCommGraph
{
  public:
    enum TypeGraph
    {
        INITIAL_MIGRATE,
        COVERAGE,
        DOF_BASED
    };
    virtual ~ParCommGraph();

    /**
     * \brief collective constructor, will be called on all sender tasks and receiver tasks
     * \param[in]  joincomm  joint MPI communicator that covers both sender and receiver MPI groups
     * \param[in]  group1   MPI group formed with sender tasks; (sender usually loads the mesh in a
     * migrate scenario) \param[in]  group2   MPI group formed with receiver tasks; (receiver
     * usually receives the mesh in a migrate scenario) \param[in]  coid1    sender component unique
     * identifier in a coupled application (in climate simulations could be the Atmosphere Comp id =
     * 5 or Ocean Comp ID , 17) \param[in]  coid2    receiver component unique identifier in a
     * coupled application (it is usually the coupler, 2, in E3SM)
     *
     * this graph will be formed on sender and receiver tasks, and, in principle, will hold info
     * about how the local entities are distributed on the other side
     *
     * Its major role is to help in migration of data, from component to the coupler and vice-versa;
     * Each local entity has a corresponding task (or tasks) on the other side, to where the data
     * needs to be sent
     *
     * important data stored in ParCommGraph, immediately it is created
     *   - all sender and receiver tasks ids, with respect to the joint communicator
     *   - local rank in sender and receiver group (-1 if not part of the respective group)
     *   - rank in the joint communicator (from 0)
     */
    ParCommGraph( MPI_Comm joincomm, MPI_Group group1, MPI_Group group2, int coid1, int coid2 );

    /**
     * \brief copy constructor will copy only the senders, receivers, compid1, etc
     */
    ParCommGraph( const ParCommGraph& );

    /**
      \brief  Based on the number of elements on each task in group 1, partition for group 2,
   trivially

   Operations: it is called on every receiver task; decides how are all elements distributed

    Note:  establish how many elements are sent from each task in group 1 to tasks in group 2
          This call is usually made on a root / master process, and will construct local maps that
   are member data, which contain the communication graph, in both directions Also, number of
   elements migrated/exchanged between each sender/receiver

     \param[in]  numElemsPerTaskInGroup1 (std::vector &)  number of elements on each sender
   task
     */

    ErrorCode compute_trivial_partition( std::vector< int >& numElemsPerTaskInGroup1 );

    /**
       \brief  pack information about receivers view of the graph, for future sending to receiver
      root

      Operations: Local, called on root process of the senders group

       \param[out] packed_recv_array
         packed data will be sent to the root of receivers, and distributed from there, and
           will have this information, for each receiver, concatenated
         receiver 1 task, number of senders for receiver 1, then sender tasks for receiver 1,
      receiver 2 task, number of senders for receiver 2, sender tasks for receiver 2, etc Note: only
      the root of senders will compute this, and send it over to the receiver root, which will
        distribute it over each receiver; We do not pack the sizes of data to be sent, only the
      senders for each of the receivers (could be of size O(n^2) , where n is the number of tasks ;
      but in general, it should be more like O(n) ). Each sender sends to a "finite" number of
      receivers, and each receiver receives from a finite number of senders). We need this info to
      decide how to set up the send/receive waiting game for non-blocking communication )
     */
    ErrorCode pack_receivers_graph( std::vector< int >& packed_recv_array );

    // get methods for private data
    bool is_root_sender()
    {
        return rootSender;
    }

    bool is_root_receiver()
    {
        return rootReceiver;
    }

    int sender( int index )
    {
        return senderTasks[index];
    }

    int receiver( int index )
    {
        return receiverTasks[index];
    }

    int get_component_id1()
    {
        return compid1;
    }
    int get_component_id2()
    {
        return compid2;
    }

    int get_context_id()
    {
        return context_id;
    }
    void set_context_id( int other_id )
    {
        context_id = other_id;
    }

    EntityHandle get_cover_set()
    {
        return cover_set;
    }
    void set_cover_set( EntityHandle cover )
    {
        cover_set = cover;
    }

    // return local graph for a specific task
    ErrorCode split_owned_range( int sender_rank, Range& owned );

    ErrorCode split_owned_range( Range& owned );

    ErrorCode send_graph( MPI_Comm jcomm, std::vector< char >& zBuff );

    ErrorCode send_graph_partition( ParallelComm* pco, MPI_Comm jcomm, std::vector< char >& zoltanBuffer );

    ErrorCode send_mesh_parts( MPI_Comm jcomm, ParallelComm* pco, Range& owned );

    // this is called on receiver side
    ErrorCode receive_comm_graph( MPI_Comm jcomm,
                                  ParallelComm* pco,
                                  std::vector< int >& pack_array,
                                  std::vector< char >& zoltanBuffer );

    ErrorCode receive_mesh( MPI_Comm jcomm,
                            ParallelComm* pco,
                            EntityHandle local_set,
                            std::vector< int >& senders_local );

    ErrorCode release_send_buffers();

    ErrorCode send_tag_values( MPI_Comm jcomm, ParallelComm* pco, Range& owned, std::vector< Tag >& tag_handles );

    ErrorCode receive_tag_values( MPI_Comm jcomm, ParallelComm* pco, Range& owned, std::vector< Tag >& tag_handles );

    // getter method
    const std::vector< int >& senders()
    {
        return senderTasks;
    }  // reference copy; refers to sender tasks in joint comm
    const std::vector< int >& receivers()
    {
        return receiverTasks;
    }

    ErrorCode settle_send_graph( TupleList& TLcovIDs );

    // this will set after_cov_rec_sizes
    void SetReceivingAfterCoverage(
        std::map< int, std::set< int > >& idsFromProcs );  // will make sense only on receivers, right now after cov

    // strideComp is np x np, or 1, in our cases
    // will fill up ordered lists for corresponding IDs on the other component
    // will form back and forth information, from ordered list of IDs, to valuesComp
    void settle_comm_by_ids( int comp, TupleList& TLBackToComp, std::vector< int >& valuesComp );

    // after map read, we need to know what entities we need to send to receiver
    ErrorCode set_split_ranges( int comp,
                                TupleList& TLBackToComp1,
                                std::vector< int >& valuesComp1,
                                int lenTag,
                                Range& ents_of_interest,
                                int type );

    // new methods to migrate mesh after reading map
    ErrorCode form_tuples_to_migrate_mesh( Interface* mb, TupleList& TLv, TupleList& TLc, int type, int lenTagType1 );
    ErrorCode form_mesh_from_tuples( Interface* mb,
                                     TupleList& TLv,
                                     TupleList& TLc,
                                     int type,
                                     int lenTagType1,
                                     EntityHandle fset,
                                     Range& primary_ents,
                                     std::vector< int >& values_entities );

    // new partition calculation
    ErrorCode compute_partition( ParallelComm* pco, Range& owned, int met, std::vector< char >& zoltanBuffer );

    // dump local information about graph
    ErrorCode dump_comm_information( const std::string & prefix, int is_send, int verbose );

  private:
    /**
    \brief find ranks of a group with respect to an encompassing communicator

    Operations: Local, usually called on root process of the group

    \param[in]  joincomm (MPI_Comm)
    \param[in]  group (MPI_Group)
    \param[out] ranks ( std::vector)  ranks with respect to the joint communicator
  */
    void find_group_ranks( MPI_Group group, MPI_Comm join, std::vector< int >& ranks );

    MPI_Comm comm;
    std::vector< int > senderTasks;    // these are the sender tasks in joint comm
    std::vector< int > receiverTasks;  // these are all the receiver tasks in joint comm
    bool rootSender;
    bool rootReceiver;
    int rankInGroup1, rankInGroup2;  // group 1 is sender, 2 is receiver
    int rankInJoin, joinSize;
    int compid1, compid2;
    int context_id;          // used to identify the other comp for intersection
    EntityHandle cover_set;  // will be initialized only if it is the receiver parcomm graph, in
                             // CoverageGraph

    // communication graph from group1 to group2;
    //  graph[task1] = vec1; // vec1 is a stl vector of tasks in group2
    std::map< int, std::vector< int > > recv_graph;  // to what tasks from group2 to send  (actual communication graph)
    std::map< int, std::vector< int > >
        recv_sizes;  // how many elements to actually send from a sender task to receiver tasks
    std::map< int, std::vector< int > >
        sender_graph;  // to what tasks from group2 to send  (actual communication graph)
    std::map< int, std::vector< int > >
        sender_sizes;  // how many elements to actually send from a sender task to receiver tasks

    std::vector< ParallelComm::Buffer* > localSendBuffs;  // this will store the pointers to the Buffers
    //                                    will be  released only when all mpi requests are waited
    //                                    for
    std::vector< int > comm_graph;  // this will store communication graph, on sender root, sent by nonblocking
                                    // send to the master receiver;  first integer will be the size of the graph,
                                    // the rest will be the packed graph, for trivial partition or Zoltan partition
                                    // the last integer will be a semaphore now that signals if we will send another
                                    // array, for the Zoltan buffer that stores right now the RCB tree
    std::vector< char > zBuff;  // this will keep the Zoltan buffer to not go out of scope before we are done sending
    //

    // these will be now used to store ranges to be sent from current sender to each receiver in
    // joint comm
    std::map< int, Range > split_ranges;

    std::vector< MPI_Request > sendReqs;  // there will be multiple requests, 2 for comm graph, 2 for each Buffer
    // there are as many buffers as sender_graph[rankInJoin].size()

    // active on both receiver and sender sides
    std::vector< int > corr_tasks;  // subset of the senderTasks, in the joint comm for sender;
                                    // subset of receiverTasks for receiver side
    std::vector< int > corr_sizes;  // how many primary entities corresponding to the other side
    // so what we know is that the local range corresponds to remote corr_sizes[i] size ranges on
    // tasks corr_tasks[i]

    // these will be used now after coverage, quick fix; they will also be populated by
    // iMOAB_CoverageGraph
    TypeGraph graph_type;  // this should be false , set to true in settle send graph, to use send_IDs_map
    std::map< int, std::vector< int > > involved_IDs_map;  // replace send and recv IDs_mapp with involved_IDs_map
    // used only for third method: DOF_BASED
    std::map< int, std::vector< int > >
        map_index;  // from index in involved[] to index in values[] of tag, for each corr task
    std::map< int, std::vector< int > > map_ptr;  //  lmap[ie], lmap[ie+1], pointer into map_index[corrTask]
};

}  // namespace moab
#endif /* SRC_PARALLEL_MOAB_PARCOMMGRAPH_HPP_ */