CONCURRENCY AND COMPUTATION: PRACTICE AND EXPERIENCE Concurrency Computat.: Pract. Exper. 2005; 17:1271–1316 Published online in Wiley InterScience (www.interscience.wiley.com). DOI: 10.1002/cpe.893 Architectural specification for massively parallel computers: an experience and measurement-based approach ‡ Ron Brightwell 1, ∗,† , William Camp 1 , Benjamin Cole 1 , Erik DeBenedictis 1 , Robert Leland 1 , James Tomkins 1 and Arthur B. Maccabe 2 1 Sandia National Laboratories, Scalable Computer Systems, P.O. Box 5800, Albuquerque, NM 87185-1110, U.S.A. 2 Department of Computer Science, University of New Mexico, Albuquerque, NM 87131-0001, U.S.A. SUMMARY In this paper, we describe the hardware and software architecture of the Red Storm system developed at Sandia National Laboratories. We discuss the evolution of this architecture and provide reasons for the different choices that have been made. We contrast our approach of leveraging high-volume, mass-market commodity processors to that taken for the Earth Simulator. We present a comparison of benchmarks and application performance that support our approach. We also project the performance of Red Storm and the Earth Simulator. This projection indicates that the Red Storm architecture is a much more cost-effective approach to massively parallel computing. Published in 2005 by John Wiley & Sons, Ltd. KEY WORDS: massively parallel computing; supercomputing; commodity processors; vector processors; Amdahl; shared memory; distributed memory 1. INTRODUCTION In the early 1980s the performance of commodity microprocessors reached a level that made it feasible to consider aggregating large numbers of them into a massively parallel processing (MPP) computer intended to compete in performance with traditional vector supercomputers based on moderate numbers of custom processors. ∗ Correspondence to: Ron Brightwell, Sandia National Laboratories, Scalable Computer Systems, P.O. Box 5800, Albuquerque, NM 87185-1110, U.S.A. † E-mail: rbbrigh@sandia.gov ‡ This article is a U.S. Government work and is in the public domain in the U.S.A. Contract/grant sponsor: U.S. Department of Energy’s National Nuclear Security Administration; contract/grant number: DE-AC04-94AL85000 Published in 2005 by John Wiley & Sons, Ltd. Received 18 February 2003 Revised 6 November 2003 Accepted 10 November 2003