Dynamic Verification of MPI Programs with Reductions in Presence of Split Operations and Relaxed Orderings ⋆ Sarvani Vakkalanka Ganesh Gopalakrishnan Robert M. Kirby School of Computing, University of Utah, Salt Lake City UT 84112, USA, http://www.cs.utah.edu/formal verification/cav08 Abstract. Dynamic verification methods are the natural choice for de- bugging real world programs when model extraction and maintenance are expensive. Message passing programs written using the MPI library fall under this category. Partial order reduction can be very effective for MPI programs because for each process, all its local computational steps, as well as many of its MPI calls, commute with the corresponding steps of all other processes. However, when dependencies arise among MPI calls, they are often a function of the runtime state. While this suggests the use of dynamic partial order reduction (DPOR), three aspects of MPI make previous DPOR algorithms inapplicable: (i) many MPI calls are allowed to complete out of program order; (ii) MPI has global synchro- nization operations (e.g., barrier) that have a special weak semantics; and (iii) the runtime of MPI cannot, without intrusive modifications, be forced to pursue a specific interleaving because of MPI’s liberal message matching rules, especially pertaining to ‘wildcard receives’. We describe our new dynamic verification algorithm ‘POE’ that exploits the out of order completion semantics of MPI by delaying the issuance of MPI calls, issuing them only according to the formation of match-sets, which are ample ‘big-step’ moves. POE guarantees to manifest any feasible in- terleaving by dynamically rewriting wildcard receives by specific-source receives. This is the first dynamic model-checking algorithm with reduc- tions for (a large subset of) MPI that guarantees to catch all deadlocks and local assertion violations, and is found to work well in practice. 1 Introduction MPI [1] programs are an important class of concurrent programs used for the distributed programming of virtually all high performance computing clusters in the world. MPI will also be widely used for programming peta-scale super- computers under construction [2]. Typical MPI programs are C programs (or C++/Fortran programs) that create a fixed number of processes at inception. These processes then perform computations in their private stores, invoking var- ious flavors of send and receive API functions in the MPI library to exchange data, and also invoke global synchronization operations in the MPI library. Most MPI programs create processes that eventually terminate. ⋆ Supported in part by NSF award CNS00509379, Microsoft HPC Institute Program, and SRC Contract 2005-TJ-1318.