Optimizing File Access Patterns through the Spatio-Temporal Pipeline for Parallel Visualization and Analysis Boonthanome Nouanesengsy, John Patchett, James Ahrens, Andrew Bauer, Aashish Chaudhary, Berk Geveci, Ross Miller, Galen M. Shipman, and Dean N. Williams Abstract—As computational resources have become more powerful over time, availability of large-scale data has exploded, with datasets greatly increasing their spatial and temporal resolutions. For many years now, I/O read time has been recognized as the primary bottleneck for parallel visualization and analysis of large-scale data. Read times ultimately depends on how the file is stored and the file access pattern used to read the file. In this paper, we introduce a model which can estimate the read time for a file stored in a parallel filesystem when given the file access pattern. The type of parallel decomposition used directly dictates what the file access pattern will be. The spatio-temporal pipeline is used to give greater flexibility to the file access pattern used. The spatio-temporal pipeline combines both spatial and temporal parallelism to create a parallel decomposition for a task. Within the spatio-temporal pipeline, all available processes are divided into groups called time compartments. Temporal parallelism is utilized as different timesteps are independently processed by separate time compartments, and spatial parallelism is used to divide each timestep over all processes within a time compartment. The ratio between spatial and temporal parallelism is controlled by adjusting the size of a time compartment. Using the model, we were able to configure the spatio-temporal pipeline to create optimized read access patterns, resulting in a speedup factor of approximately 400 over traditional file access patterns. Index Terms—Visualization, Data Analysis, I/O, Modeling, Parallel Techniques 1 I NTRODUCTION The visualization and analysis of large-scale data in a timely manner has been a recognized problem for many years now. With computa- tional power increasing and the introduction of more sensitive instru- mentation, data from both simulations and experiments are expected to continue to grow. The result of these trends are ever larger datasets containing higher spatial and temporal resolutions. The standard re- sponse to tackle the large-data problem is to use parallel processing capabilities. The main bottleneck in large-scale parallel analysis is usually the I/O read step. One of the main factors in I/O performance is how the file is accessed, and what the read pattern is. The parallel decomposition strategy determines the read pattern. For many of the common visualization tools used by the commu- nity, parallel processing implies using a data-parallel approach em- ploying only spatial parallelism. In this approach, the data is parti- tioned spatially and spread out across several processes. Each process then applies the same computation on their piece of data. Another op- tion for data decomposition, temporal parallelism, involves processing data from different timesteps simultaneously. The same computations are applied to each timestep. How different decomposition approaches • Boonthanome Nouanesengsy is with Los Alamos National Laboratory. E-mail: boonth@lanl.gov. • John Patchett is with Los Alamos National Laboratory. E-mail: patchett@lanl.gov. • James Ahrens is with Los Alamos National Laboratory. E-mail: ahrens@lanl.gov. • Andrew Bauer is with Kitware, Inc.. E-mail: andy.bauer@kitware.com. • Aashish Chaudhary is with Kitware, Inc.. E-mail: aashish.chaudhary@kitware.com. • Berk Geveci is with Kitware, Inc.. E-mail: berk.geveci@kitware.com. • Ross Miller is with Oak Ridge National Laboratory. E-mail: rgmiller@ornl.gov. • Galen M. Shipman is with Oak Ridge National Laboratory. E-mail: gshipman@ornl.gov. • Dean N. Williams is with Lawrence Livermore National Laboratory. E-mail: williams13@llnl.gov. Manuscript received 31 March 2013; accepted 1 August 2013; posted online 13 October 2013; mailed on 27 September 2013. For information on obtaining reprints of this article, please send e-mail to: tvcg@computer.org. affect the pattern used to read in files is often overlooked, despite the fact that I/O performance is usually the performance bottleneck. In this paper, we describe a model which can estimate the time needed to load a file stored in a parallel filesystem when given the file access pattern. Using this model, we can determine which parallel approaches are best suited for a given file format and pipeline. To offer more control over the file access pattern, we have developed a method that employs both spatial and temporal parallelism, which we call the spatio-temporal pipeline. In the spatio-temporal pipeline, all available processes are separated into groups called time compartments. Tempo- ral parallelism is employed as each time compartment independently processes one or several timesteps concurrently. Within a time com- partment, a timestep is partitioned spatially over all processes within the time compartment, thus the spatio-temporal pipeline uses both spa- tial and temporal parallelism. The spatio-temporal pipeline is a fully integrated feature in the customized version of ParaView released with UV-CDAT (Ultrascale Visualization - Climate Data Analysis Tools). UV-CDAT [14] is a visualization and analysis tool specializing in large-scale climate-data analysis. Time-varying data can be stored in a myriad of different formats. For example, all data over all timesteps can be stored in one master file. Another possibility is to have separate spatial partitions stored in different files, while each file spans multiple timesteps. How the data is stored and what access patterns are used to read in the data play a critical role in I/O read performance. In this paper, we focus our discussion to a very common file format: the time-varying data is composed of files, and each file represents one scalar field at one timestep. The benefits of temporal parallelism versus spatial parallelism may not be readily apparent. Indeed, if all components of a parallel system were to scale perfectly, there should be no difference in running time between using spatial parallelism, temporal parallelism, and spatio- temporal parallelism, for equal amounts of parallelization. This is be- cause the same amount of work is being processed by equal amounts of hardware. Ultimately, using spatio-temporal parallelism allows for greater flexibility in selecting a file access pattern that will provide greater I/O performance. The model is used to help determine which file access patterns should be used, as well as helping to determine parameter values such as time compartment size. In this paper, we contribute a model which estimates the total run- ning time of a parallel visualization and analysis pipeline, including