Mpi

MPI

MPI is the message-passing standard for distributed-memory parallel programs: one process per rank, explicit sends, receives, and collectives over a communicator. The commands below are exercised against Open MPI 5.0.10, the implementation on the grounding host; MPICH and vendor MPIs accept the same API and a different mpirun option set.

Contract

Inputs

• MPI implementation: mpicc --version and mpirun --version name it. mpicc wraps the C compiler with the include and link flags.
• Rank count and placement: cores per node, sockets, and whether OpenMP threads share a rank.
• For multi-node runs: a hostfile (node0 slots=4) or a scheduler allocation (SLURM_NTASKS).
• The failing symptom, when debugging: hang, wrong result, MPI_ERR_TRUNCATE, or poor scaling.

Procedure

1. Start from the minimal program and prove the launch path:

   #include <mpi.h>
   #include <stdio.h>
   
   int main(int argc, char **argv) {
       MPI_Init(&argc, &argv);
       int rank, size;
       MPI_Comm_rank(MPI_COMM_WORLD, &rank);
       MPI_Comm_size(MPI_COMM_WORLD, &size);
       printf("rank %d of %d\n", rank, size);
       MPI_Finalize();
       return 0;
   }
   

   mpicc -O2 -o hello hello.c then mpirun -np 4 ./hello (mpiexec -n 4 ./hello is the standard's spelling). On a laptop with fewer cores than ranks add --oversubscribe. Done when: every rank prints once.

2. Point-to-point. MPI_Send(buf, count, MPI_INT, dest, tag, comm) pairs with MPI_Recv(buf, count, MPI_INT, source, tag, comm, MPI_STATUS_IGNORE); a receive matches on communicator, source, and tag, and MPI_ANY_SOURCE or MPI_ANY_TAG widen it. A receive count smaller than the incoming message raises MPI_ERR_TRUNCATE; larger is fine and MPI_Get_count on the status reports the real length. Done when: every send has a receive with the same communicator, tag, and datatype, and the receive buffer is at least as large.

3. Collectives instead of hand-rolled loops:

   Call	Purpose
   MPI_Bcast	One rank's buffer to all
   MPI_Scatter, MPI_Gather	Split a root buffer into chunks, or collect chunks
   MPI_Allreduce(&local, &global, 1, MPI_INT, MPI_SUM, comm)	Reduce and deliver the result to every rank
   MPI_Reduce	Reduce to the root only
   MPI_Alltoall	Every rank sends a chunk to every rank
   MPI_Barrier	Synchronize; use it for timing, not for correctness

   Every rank in the communicator must call the collective with matching datatype and count. Done when: no collective is emulated with sends and receives, and the datatypes match on all ranks.

4. Overlap communication with computation using non-blocking requests. MPI_Isend and MPI_Irecv return an MPI_Request; do local work; then MPI_Wait(&req, MPI_STATUS_IGNORE) or MPI_Waitall(n, reqs, MPI_STATUSES_IGNORE). The buffer belongs to MPI until the wait returns. Post receives before the matching sends where possible, so messages land in user buffers instead of internal ones. Done when: every request is completed by a wait or test before its buffer is reused or freed.

5. Split communicators for groups. MPI_Comm_split(MPI_COMM_WORLD, color, key, &subcomm) puts ranks with equal color in one communicator ordered by key; free it with MPI_Comm_free. Done when: group-local collectives run on the subcommunicator and it is freed.

6. Hybrid MPI and OpenMP. Initialize with MPI_Init_thread(&argc, &argv, MPI_THREAD_FUNNELED, &provided) when only the main thread calls MPI, or MPI_THREAD_MULTIPLE when any thread does, and check provided. Size the threads to the cores per rank: OMP_NUM_THREADS = cores_per_node / ranks_per_node. Place ranks with Open MPI's --map-by ppr:2:package --bind-to core (socket is the deprecated spelling of package); --report-bindings prints the result. openmp covers the thread side. Done when: ranks times threads equals the cores in use and --report-bindings shows no overlap.

7. Launch on several nodes. Hostfile lines are node0 slots=4; run mpirun -np 8 --hostfile hosts.txt ./app. Under Slurm, srun -n 64 ./app or mpirun -np $SLURM_NTASKS ./app inside the allocation. --tag-output prefixes each output line with its rank. Done when: ranks land on the intended nodes.

8. MPI-IO for parallel files:

   MPI_File fh;
   MPI_File_open(MPI_COMM_WORLD, "output.dat", MPI_MODE_CREATE | MPI_MODE_WRONLY, MPI_INFO_NULL, &fh);
   MPI_Offset offset = (MPI_Offset)rank * chunk_bytes;
   MPI_File_write_at_all(fh, offset, buf, count, MPI_DOUBLE, MPI_STATUS_IGNORE);
   MPI_File_close(&fh);
   

   The _all collective form lets the library aggregate requests; MPI_File_write_at is the independent form. Done when: every rank writes its own offset range and the file size equals ranks times chunk.

9. Diagnose scaling. Load imbalance: measure per-rank time with MPI_Wtime around the work phase and redistribute chunks. Serialization at rank 0: replace gather-then-process with a reduction tree (MPI_Reduce, MPI_Allreduce) and parallel I/O. Excess synchronization: remove barriers that guard nothing. Many small messages: aggregate into larger buffers or derived datatypes. MPI_Alltoall at large rank counts: use neighborhood collectives (MPI_Neighbor_alltoall) on a topology communicator when each rank talks to few peers. Profile rank hotspots with linux-perf (perf record on one rank's PID). Done when: the bottleneck is named with a measurement and the fix is applied or proposed.

Failure and recovery

Output

An MPI program that compiles with mpicc, runs under mpirun with correct output per rank, has matched sends and receives and completed requests, and, when performance was the question, a named bottleneck with its measurement and the applied fix; plus the launch command with its placement flags.