IEEE TRANSACTIONS ON NUCLEAR SCIENCE, VOL. 55, NO. 3, JUNE 2008 1621 A Comparison of Data-Access Platforms for the Computing of Large Hadron Collider Experiments M. Bencivenni, F. Bonifazi, A. Carbone, A. Chierici, A. D’Apice, D. De Girolamo, L. dell’Agnello, M. Donatelli, G. Donvito, A. Fella, F. Furano, D. Galli, A. Ghiselli, A. Italiano, G. Lo Re, U. Marconi, B. Martelli, M. Mazzucato, M. Onofri, P. P. Ricci, F. Rosso, D. Salomoni, V. Sapunenko, V. Vagnoni, R. Veraldi, M. C. Vistoli, D. Vitlacil, and S. Zani Abstract—Performance, reliability and scalability in data-ac- cess are key issues in the context of the computing Grid and High Energy Physics data processing and analysis applications, in particular considering the large data size and I/O load that a Large Hadron Collider data centre has to support. In this paper we present the technical details and the results of a large scale validation and performance measurement employing different data-access platforms—namely CASTOR, dCache, GPFS and Scalla/Xrootd. The tests have been performed at the CNAF Tier-1, the central computing facility of the Italian National Institute for Nuclear Research (INFN). Our storage back-end was based on Fibre Channel disk-servers organized in a Storage Area Net- work, being the disk-servers connected to the computing farm via Gigabit LAN. We used 24 disk-servers, 260 TB of raw-disk space and 280 worker nodes as computing clients, able to run concurrently up to about 1100 jobs. The aim of the test was to perform sequential and random read/write accesses to the data, as well as more realistic access patterns, in order to evaluate efficiency, availability, robustness and performance of the various data-access solutions. Index Terms—Computer facilities, computer input-output, data handling, data processing, mass memories, system analysis and de- sign. I. INTRODUCTION T HE WORLD-WIDE LHC COMPUTING GRID (WLCG) [1], i.e. the distributed computing infrastructure for the ex- periments at the Large Hadron Collider (LHC) [2], [23], [24], has the goal of processing several petabytes (PB) of data per year produced by the LHC experiments. In order to cope with such a large amount of data, the WLCG model foresees distri- bution of the data to a plethora of computing centres around the world, organized in a hierarchical structure, the so-called Tier-1 and Tier-2 centres. The data need to be exchanged between Manuscript received December 7, 2007; revised April 1, 2008. M. Bencivenni, F. Bonifazi, A. Carbone, A. Chierici, A. D’Apice, D. De Giro- lamo, L. dell’Agnello, M. Donatelli, A. Fella, A. Ghiselli, A. Italiano, G. Lo Re, B. Martelli, M. Mazzucato, M. Onofri, P. P. Ricci, F. Rosso, D. Salomoni, V. Sapunenko, R. Veraldi, M. C. Vistoli, D. Vitlacil, and S. Zani are with the Istituto Nazionale di Fisica Nucleare, CNAF, I-40126 Bologna, Italy (e-mail: luca.dellagnello@cnaf.infn.it). G. Donvito is with the Istituto Nazionale di Fisica Nucleare, Sezione di Bari, I-70126 Bari, Italy. F. Furano is with the Istituto Nazionale di Fisica Nucleare, Sezione di Padova, I-35131 Padova, Italy. D. Galli, U. Marconi, and V. Vagnoni are with the Istituto Nazionale di Fisica Nucleare, Sezione di Bologna, I-40126 Bologna, Italy. Digital Object Identifier 10.1109/TNS.2008.924087 CERN and all the national computing centres, where the pro- cessing phase will take place. The data transfer and processing require steady I/O throughputs of tens of Gb/s, hence posing very stringent requirements for all the data centres involved. To satisfy such requirements, the computing centres must employ very large, reliable and powerful storage infrastructures. In this paper we present a comparative evaluation of some data-access solutions available nowadays, namely the CERN Advanced Storage Manager (CASTOR) [3], the dCache system [4], the General Parallel File System (GPFS) [5] and the Scalla/ Xrootd system [6]. At the time of writing all these platforms were free of license fees for educational and research activities. These systems are already installed at many Tier-1 and Tier-2 centres, but a large scale comparative test of each of them em- ploying the same hardware infrastructure, in order to obtain a meaningful comparison, was still missing. The aim of our work is then to provide the INFN-CNAF Tier-1 centre, as well as other similar large or smaller computing sites, with comparative fig- ures of merit useful for finalizing the technical choices of the storage systems, before the startup of the LHC operation, ex- pected by the end of 2008. We have tested different kinds of data-access patterns, ranging from purely sequential up to fully random ones. We also emulated the data analysis access pattern of a running High Energy Physics (HEP) experiment, namely BaBar [7], by using a technique based on trace files. Trace files were obtained by sampling all the I/O system calls of a real BaBar analysis program and writing template files containing, for each call, a couple of integer numbers representing a pointer to the position of the data block to be accessed in the data file (offset), together with the size of the data block. These template files were then used as inputs to our custom applications, with the aim of testing the storage systems under a realistic access pattern. In order to fully simulate the behavior of the data anal- ysis applications, we have also taken into account the impact of consumed CPU cycles by running dummy computations between subsequent I/O calls. Finally, real analysis jobs from the LHCb Collaboration [8] have been comparatively run on the same data sets served by the different data-access systems. In Section II we will briefly describe the current production storage infrastructure at CNAF. In Section III we will introduce the layout and tuning of our test-bed, the data-access platforms we have evaluated and the monitoring tools we have been using. Finally, in Section IV we will discuss the tests on the various data-access platforms and the results we obtained. 0018-9499/$25.00 © 2008 IEEE