Extended Collective Operation[MPI2]

Extended Collective Operation

Introduction

MPI-1에서는 intrecommnunicator를 위한 collective communication과 두 가지 routine

즉, 새로운 intracommunicator를 생성하기 위해서 MPI_INTERCOMM_CREATE와 MPI_COMM_DUP를

제공하고 있다. 그리고 Fortran과의 aliasing 문제를 피하기 위해서 collective operation을 위한

send 와 receive buffer를 각각 유지하는 것이 필요했다.

MPI-2에서는 이러한 intracommunicator에 대한 collective routine과 intercommunicator를 생성하기 위한

routine 그리고 generalized all-to-all과 exclusive scan이라는 routine을 확장했다. 많은 intracommunicator

collective routine에 대한 적절한 버퍼를 정의하는 방법을 제공하고 있다.

Intercommunicator Constructors

현재 MPI interface에는 단지 intercommunicator construction routine 을 제공하고 있다.

MPI_INTERCOMM_CREATE : 두 개의 intracommunicator에 대한 intercommunicator 생성

MPI_COMM_DUP : 존재하는 intercommunicator ( or intracommunicator ) 를 복사

MPI_COMM_CREATE와 MPI_COMM_SPLIT는 현재 intracommunicator에만 적용된다.

MPI_COMM_CREATE(comm_in, group, comm_out)

[ IN comm_in] original communicator (handle)

[ IN group] group of processes to be in new communicator (handle)

[ OUT comm_out] new communicator (handle)

Group argument는 comm_out이 될 input intercommunicator의 local group의 process들만을 포함하고 있어야 한다.

만일 어떤 group 이 intercommunicator의 local group 에서 적어도 하나의 process를 포함하고 있지 않다면 또는

group에서 calling process가 포함되어 있지 않다면 MPI_COMM_NULL을 리턴한다.

MPI_Comm inter_comm, new_inter_comm;

MPI_Group local_group, group;

int rank = 0; /* rank on left side to include in

new inter-comm */

/* Construct the original intercommunicator: "inter_comm" */

...

/* Construct the group of processes to be in new

intercommunicator */

if (/* I'm on the left side of the intercommunicator */) {

MPI_Comm_group ( inter_comm, &local_group );

MPI_Group_incl ( local_group, 1, &rank, &group );

MPI_Group_free ( &local_group );

}

else

MPI_Comm_group ( inter_comm, &group );

MPI_Comm_create ( inter_comm, group, &new_inter_comm );

MPI_Group_free( &group );

MPI_COMM_SPLIT(comm_in, color, key, comm_out)

[ IN comm_in] original communicator (handle)

[ IN color] control of subset assignment (integer)

[ IN key] control of rank assignment (integer)

[ OUT comm_out] new communicator (handle)

Intercommunicator에서 MPI_COMM_SPLIT의 결과는 같은 color의 process들로 새로운 intercommunicator 를 만드는 것이다.

Key value는 각각의 상대 rank를 나타낸다. 만일 color이 한쪽에 정의되어 있지 않다면 MPI_COMM_NULL이 리턴되고 color가

MPI_UNDEFINED로 정의된 process에도 MPI_COMM_NULL이 리턴된다.

/* Client code */

MPI_Comm multiple_server_comm;

MPI_Comm single_server_comm;

int color, rank, num_servers;

/* Create intercommunicator with clients and servers:

multiple_server_comm */

...

/* Find out the number of servers available */

MPI_Comm_remote_size ( multiple_server_comm, &num_servers );

/* Determine my color */

MPI_Comm_rank ( multiple_server_comm, &rank );

color = rank % num_servers;

/* Split the intercommunicator */

MPI_Comm_split ( multiple_server_comm, color, rank,

&single_server_comm );

The following is the corresponding server code:

/* Server code */

MPI_Comm multiple_client_comm;

MPI_Comm single_server_comm;

int rank;

/* Create intercommunicator with clients and servers:

multiple_client_comm */

...

/* Split the intercommunicator for a single server per group

of clients */

MPI_Comm_rank ( multiple_client_comm, &rank );

MPI_Comm_split ( multiple_client_comm, rank, 0,

&single_server_comm );

Extended Collective Operations

기존의 MPI-1에서의 collective operation은 단지 intracommunicator에만 적용이 되었지만 이것은 intercommunicator로 그대로 확장이 될 수가 있다.

All-To-All : 모든 process가 결과에 참가를 하고 그 결과를 받게 된다.

MPI_Allgather, MPI_Allgatherv

MPI_Alltoall, MPI_Alltoallv

MPI_Allreduce, MPI_Reduce_scatter

All-To-One : 모든 process가 결과에 참가를 하지만 한 process만 그 결과를 받는다.

MPI_Gather, MPI_Gatherv

MPI_Reduce

One-To-All : 한 process만 결과에 참가를 하고 모든 process가 그 결과를 받는다.

MPI_Bcast

MPI_Scatter, MPI_Scatterv

Other : 위의 경우에 해당하지 않는 Collective operation

MPI_Scan

MPI_Barrier

Example

All-to-all : 한group의 모든 멤버로부터 데이터를 모아서 다른 group의 멤버에서 결과가 나

타난다.

One-to-all : 어떤 group의 한 멤버로부터 데이터를 받아서 다른 group의 모든 멤버에게 보낸

다. 이것은 MPI_REDUCE_SCATTER과 같다.

Operations that Move Data

많은 collective communication call에 두 가지 추가된 것이 있다.

collective communication 는 intracommunication에 대해서 “in-place”를 해줌으로써 output buffer 를

input buffer와 같도록 해줄 수가 있다. send buffer 와 receive buffer의 argument 에 MPI_IN_PLACE를 주면 된다.

만일 operation 이 rooted 즉, broadcast, gather,scatter 같은 것이면 unidirectional 하고 unrooted라면 bidirectional 하다.

Broadcast

MPI_BCAST(buffer, count, datatype, root, comm)

[ INOUT buffer] starting address of buffer (choice)

[ IN count] number of entries in buffer (integer)

[ IN datatype] data type of buffer (handle)

[ IN root] rank of broadcast root (integer)

[ IN comm] communicator (handle)

comm이 intercommunicator라면 intercommunicator의 모든 process가 되고 만일 group라면 root process가 된다.

Root는 root에 MPI_ROOT를 다른 것은 MPI_PROC_NULL을 pass한다.

Gather

MPI_GATHER(sendbuf, sendcount, sendtype, recvbuf, recvcount, recvtype, root, comm)

[ IN sendbuf] starting address of send buffer (choice)

[ IN sendcount] number of elements in send buffer (integer)

[ IN sendtype] data type of send buffer elements (handle)

[ OUT recvbuf] address of receive buffer (choice, significant only at root)

[ IN recvcount] number of elements for any single receive (integer, significant only at root)

[ IN recvtype] data type of recv buffer elements (handle, significant only at root)

[ IN root] rank of receiving process (integer)

[ IN comm] communicator (handle)

MPI_GATHERV(sendbuf, sendcount, sendtype, recvbuf, recvcounts, displs, recvtype, root, comm)

[ IN sendbuf] starting address of send buffer (choice)

[ IN sendcount] number of elements in send buffer (integer)

[ IN sendtype] data type of send buffer elements (handle)

[ OUT recvbuf] address of receive buffer (choice, significant only at root)

[ IN recvcounts] integer array (of length group size) containing the number of elements that are received from each process (significant only at root)

[ IN displs] integer array (of length group size). Entry i specifies the displacement relative to recvbuf at which to place the incoming data from process i (significant only at root)

[ IN recvtype] data type of recv buffer elements (handle, significant only at root)

[ IN root] rank of receiving process (integer)

[ IN comm] communicator (handle)

Scatter

MPI_SCATTER(sendbuf, sendcount, sendtype, recvbuf, recvcount, recvtype, root, comm)

[ IN sendbuf] address of send buffer (choice, significant only at root)

[ IN sendcount] number of elements sent to each process (integer, significant only at root)

[ IN sendtype] data type of send buffer elements (handle, significant only at root)

[ OUT recvbuf] address of receive buffer (choice)

[ IN recvcount] number of elements in receive buffer (integer)

[ IN recvtype] data type of receive buffer elements (handle)

[ IN root] rank of sending process (integer)

[ IN comm] communicator (handle)

MPI_SCATTERV(sendbuf, sendcounts, displs, sendtype, recvbuf, recvcount, recvtype, root, comm)

[ IN sendbuf] address of send buffer (choice, significant only at root)

[ IN sendcounts] integer array (of length group size) specifying the number of elements to send to each processor

[ IN displs] integer array (of length group size). Entry i specifies the displacement (relative to sendbuf from which to take the outgoing data to process i

[ IN sendtype] data type of send buffer elements (handle)

[ OUT recvbuf] address of receive buffer (choice)

[ IN recvcount] number of elements in receive buffer (integer)

[ IN recvtype] data type of receive buffer elements (handle)

[ IN root] rank of sending process (integer)

[ IN comm] communicator (handle)

All form and All-to-all

MPI_ALLGATHER(sendbuf, sendcount, sendtype, recvbuf, recvcount, recvtype, comm)

[ IN sendbuf] starting address of send buffer (choice)

[ IN sendcount] number of elements in send buffer (integer)

[ IN sendtype] data type of send buffer elements (handle)

[ OUT recvbuf] address of receive buffer (choice)

[ IN recvcount] number of elements received from any process (integer)

[ IN recvtype] data type of receive buffer elements (handle)

[ IN comm] communicator (handle)

MPI_ALLGATHERV(sendbuf, sendcount, sendtype, recvbuf, recvcounts, displs, recvtype, comm)

[ IN sendbuf] starting address of send buffer (choice)

[ IN sendcount] number of elements in send buffer (integer)

[ IN sendtype] data type of send buffer elements (handle)

[ OUT recvbuf] address of receive buffer (choice)

[ IN recvcounts] integer array (of length group size) containing the number of elements that are received from each process

[ IN displs] integer array (of length group size). Entry i specifies the displacement (relative to recvbuf) at which to place the incoming data from process i

[ IN recvtype] data type of receive buffer elements (handle)

[ IN comm] communicator (handle)

MPI_ALLTOALL(sendbuf, sendcount, sendtype, recvbuf, recvcount, recvtype, comm)

[ IN sendbuf] starting address of send buffer (choice)

[ IN sendcount] number of elements sent to each process (integer)

[ IN sendtype] data type of send buffer elements (handle)

[ OUT recvbuf] address of receive buffer (choice)

[ IN recvcount] number of elements received from any process (integer)

[ IN recvtype] data type of receive buffer elements (handle)

[ IN comm] communicator (handle)

MPI_ALLTOALLV(sendbuf, sendcounts, sdispls, sendtype, recvbuf, recvcounts, rdispls, recvtype, comm)

[ IN sendbuf] starting address of send buffer (choice)

[ IN sendcounts] integer array equal to the group size specifying the number of elements to send to each processor

[ IN sdispls] integer array (of length group size). Entry j specifies the displacement (relative to sendbuf) from which to take the outgoing data destined for process j

[ IN sendtype] data type of send buffer elements (handle)

[ OUT recvbuf] address of receive buffer (choice)

[ IN recvcounts] integer array equal to the group size specifying the number of elements that can be received from each processor

[ IN rdispls] integer array (of length group size). Entry i specifies the displacement (relative to recvbuf) at which to place the incoming data from process i

[ IN recvtype] data type of receive buffer elements (handle)

[ IN comm] communicator (handle)

Reductions

MPI_REDUCE(sendbuf, recvbuf, count, datatype, op, root, comm)

[ IN sendbuf] address of send buffer (choice)

[ OUT recvbuf] address of receive buffer (choice, significant only at root)

[ IN count] number of elements in send buffer (integer)

[ IN datatype] data type of elements of send buffer (handle)

[ IN op] reduce operation (handle)

[ IN root] rank of root process (integer)

[ IN comm] communicator (handle)

MPI_ALLREDUCE(sendbuf, recvbuf, count, datatype, op, comm)

[ IN sendbuf] starting address of send buffer (choice)

[ OUT recvbuf] starting address of receive buffer (choice)

[ IN count] number of elements in send buffer (integer)

[ IN datatype] data type of elements of send buffer (handle)

[ IN op] operation (handle)

[ IN comm] communicator (handle)

MPI_REDUCE_SCATTER(sendbuf, recvbuf, recvcounts, datatype, op, comm)

[ IN sendbuf] starting address of send buffer (choice)

[ OUT recvbuf] starting address of receive buffer (choice)

[ IN recvcounts] integer array specifying the number of elements in result distributed to each process. Array must be identical on all calling processes.

[ IN datatype] data type of elements of input buffer (handle)

[ IN op] operation (handle)

[ IN comm] communicator (handle)

Other operations

MPI_BARRIER(comm)

[ IN comm] communicator (handle)

MPI_SCAN(sendbuf, recvbuf, count, datatype, op, comm)

[ IN sendbuf] starting address of send buffer (choice)

[ OUT recvbuf] starting address of receive buffer (choice)

[ IN count] number of elements in input buffer (integer)

[ IN datatype] data type of elements of input buffer (handle)

[ IN op] operation (handle)

[ IN comm] communicator (handle)

Generalized All-to-all Function

MPI_ALLTOALLW(sendbuf, sendcounts, sdispls, sendtypes, recvbuf, recvcounts, rdispls, recvtypes, comm)

[ IN sendbuf] starting address of send buffer (choice)

[ IN sendcounts] integer array equal to the group size specifying the number of elements to send to each processor (integer)

[ IN sdispls] integer array (of length group size). Entry j specifies the displacement in bytes (relative to sendbuf) from which to take the outgoing data destined for process j

[ IN sendtypes] array of datatypes (of length group size). Entry j specifies the type of data to send to process j (handle)

[ OUT recvbuf] address of receive buffer (choice)

[ IN recvcounts] integer array equal to the group size specifying the number of elements that can be received from each processor (integer)

[ IN rdispls] integer array (of length group size). Entry i specifies the displacement in bytes (relative to recvbuf) at which to place the incoming data from process i

[ IN recvtypes] array of datatypes (of length group size). Entry i specifies the type of data received from process i (handle)

[ IN comm] communicator (handle)

Extensive Scan

MPI_EXSCAN(sendbuf, recvbuf, count, datatype, op, comm)

[ IN sendbuf] starting address of send buffer (choice)

[ OUT recvbuf] starting address of receive buffer (choice)

[ IN count] number of elements in input buffer (integer)

[ IN datatype] data type of elements of input buffer (handle)

[ IN op] operation (handle)

[ IN comm] intracommunicator (handle)