Cluster Comput DOI 10.1007/s10586-009-0086-y Full-system simulation of distributed memory multicomputers Fco. Javier Ridruejo · Jose Miguel-Alonso · Javier Navaridas Received: 8 October 2008 / Accepted: 8 March 2009 © Springer Science+Business Media, LLC 2009 Abstract In this paper we discuss environments for the full- system simulation of multicomputers. These environments are composed of a large collection of modules that simulate the compute nodes and the network, plus additional link- ing elements that perform communication and synchroniza- tion. We present our own environment, in which we integrate Simics with INSEE. We reuse as many Simics modules as possible to reduce the effort of hardware modeling, and also to simulate standard machines running unmodified operat- ing systems. This way we avoid the error-prone effort of de- veloping drivers and libraries. The environment we propose in this paper enables us to show some of the difficulties we found when integrating diverse tools, and how we were able to overcome them. Furthermore we show some important details to have into account in order to do a valid full-system simulation of multicomputers, mostly related with synchro- nization and timing. Thus, a trade-off has to be found be- tween simulation speed and accuracy of results. Keywords Full-system simulation · Interconnection networks · Multicomputers · Clusters F.J. Ridruejo · J. Miguel-Alonso ( ) · J. Navaridas Dep. of Computer Architecture and Technology, The University of the Basque Country, P. Manuel de Lardizabal, 1, 20018 Donostia-San Sebastian, Spain e-mail: j.miguel@ehu.es F.J. Ridruejo e-mail: franciscojavier.ridruejo@ehu.es J. Navaridas e-mail: javier.navaridas@ehu.es 1 Introduction Supercomputing is a very valuable resource which is con- tinuously growing in importance in business and science. Current scientific studies rely on analysis and modeling of different natural phenomena that require a huge amount of computing power, which is not attainable using regular off- the-shelf computers. For example, physicists, chemists or pharmaceutical researchers simulate, for different purposes, interactions between large numbers of molecules. Likewise, in the business context, corporations demand large amounts of computing power in order to use data mining software over their huge data bases, with the objective of extracting knowledge from raw data, and to use that knowledge to their advantage. Obtaining patterns of consumer habits, boosting sales, optimizing costs and profits, estimating stocks or de- tecting fraudulent behavior are just a few interesting appli- cation domains. At any rate, in both contexts, the required computing power is only limited by the available resources. In general, if available computing resources are doubled the number of runs, the grain-size (whichever this means in the particular application context), the size of the datasets or whatever other parameter that affects execution time will be increased in order to fully use the new resources. In other words, the magnitude of the experiments is scaled up to the available resources. This means that there is a permanent demand of high-performance computers able to cope with these challenging workloads. A supercomputer is not only a piece of hardware. It is actually a multipart system that integrates a large collection of hardware and software elements. Therefore, the design of a supercomputer is a complex task that comprises the selec- tion and design of multiple components, such as compute el- ements, storage, interconnection network, access elements,