Pink: A 1024-node Single-System Image Linux Cluster * Gregory R. Watson Matthew J. Sottile Ronald G. Minnich Sung-Eun Choi Erik A. Hendriks Cluster Research Lab Advanced Computing Laboratory, CCS-1 Los Alamos National Laboratory † Los Alamos, NM 87545 cluster@lanl.gov Abstract This paper describes our experience of designing and building Pink, a 1024-node (2048 processor) Myrinet-based single-system image Linux cluster that was installed in Jan- uary 2003 at the Los Alamos National Laboratory. At the time of its installation, Pink was the largest single-system image Linux cluster in the world, and was based entirely on open-source software - from the BIOS up. Pink was the proof-of-concept prototype for Lightning, a production 1408-node (2816 processor) cluster that will begin opera- tion at LANL later this year. Lightning is currently number 6 on the Top500 list. In this paper we will examine the issues that were en- countered and the problems that needed to be overcome in order to scale a cluster to this size. We will also present some performance numbers that demonstrate the scalabil- ity and manageability of the cluster software suite. 1. Introduction In the past five years, Linux has gone from running on only a few small clusters to being the operating system of choice for the next-generation of high performance com- puting (HPC) systems at the Department of Energy (DOE) labs. The strong growth of Linux in the commercial sector, * This research was funded by the Mathematical Information and Com- puter Sciences (MICS) Program of the DOE Office of Science, the Los Alamos Computer Science Institute (ASCI Institutes), and the Labo- ratory Directed Research and Development (LDRD) program. LANL LA-UR-04-1890. † Los Alamos National Laboratory is operated by the University of Cal- ifornia for the National Nuclear Security Administration of the United States Department of Energy under contract W-7405-ENG-36. combined with the unique requirements of the HPC envi- ronment, has convinced DOE to invest in both the develop- ment of HPC software and the development of expertise in Linux and other open source software. As part of this effort, Los Alamos National Laboratory (LANL) has been undertaking a research project funded by the Mathematical Information and Computer Sciences (MICS) Program of the DOE Office of Science to design a fundamentally new clustering technology, known as Clus- termatic. The aim of this project is to develop a complete cluster management system with the primary goal of mak- ing clusters easier to maintain and use at virtually any scale. The 1024-node Pink cluster is the first milestone in the re- alization of this research project, which is ultimately aimed at developing a 100 Teraflops Linux cluster. Pink is also the proof-of-concept for Lightning, a 1408-node production cluster that is currently number 6 on the November 2003 Top500 [6] list. At the time of it’s installation, Pink was the hallmark of a number of important achievements in clustering technology for a machine of this size, including: • The machine was the largest single-system im- age Linux cluster in the world. • The vendor (Linux Networx) provided all the hard- ware, but almost none of the system software. • The entire software suite, including the BIOS, operat- ing system, clustering software, and other system tools were all open-source projects based at LANL and other sites. • The system did not require a second management net- work, instead the Myrinet interconnect was used for both system booting and monitoring purposes, as well as a traditional HPC application interconnect. Proceedings of the Seventh International Conference on High Performance Computing and Grid in Asia Pacific Region (HPCAsia’04) 0-7695-2138-X/04 $ 20.00 IEEE