simplifies the load-balancing, data movement, unstructured-communica- tion, and memory usage difficulties that arise in dynamic applications such as adaptive finite-element methods, parti- cle methods, and crash simulations. Zoltan’s data-structure-neutral design also lets a wide range of applications use it without imposing restrictions on ap- plication data structures. Its object- based interface provides a simple and inexpensive way for application devel- opers to use the library and researchers to make new capabilities available under a common interface. Zoltan provides tools that help appli- cation developers without imposing strict requirements on them. For exam- ple, it includes a suite of parallel parti- tioning algorithms and data migration tools that redistribute data to reflect, say, changing processor workloads. Zoltan also includes distributed data di- rectories and unstructured communica- tion services that let applications per- form complicated communication using only a few simple primitives. To simplify debugging of dynamic memory usage, Zoltan provides dynamic memory man- agement tools that enhance common memory allocation functions. In this ar- ticle, we describe Zoltan’s features and ways to use it in dynamic applications. Zoltan software design Our design of the Zoltan library does not restrict it to any particular type of application. Rather, Zoltan operates on uniquely identifiable data items that we call objects. For example, in finite-ele- ment applications, objects might be el- ements or nodes of the mesh. In parti- cle applications, objects might be particles. In linear solvers, objects might be matrix rows. Each object must have a unique global identifier (ID) represented as an array of un- signed integers. Common choices in- clude global numbers for elements (nodes, rows, particles, and so on) that already exist in many applications, or a structure consisting of an owning processor number and the object’s lo- cal-memory index. Objects might also have local (to a processor) IDs that do not have to be unique globally. Local IDs such as addresses or local-array in- dices of objects can improve the per- formance (and convenience) of Zoltan’s interface to applications. To make Zoltan easy to use, we do not impose any particular data struc- ture on an application, nor do we re- quire an application to build a particu- lar data structure for Zoltan. Instead, Zoltan uses a callback function inter- face in which the tool queries the ap- plication for needed data. The applica- tion must provide simple functions that answer these queries. For example, Figure 1 shows how Zoltan’s callback function interface works for performing dynamic load bal- ancing in an application. An application starts Zoltan (Zoltan_Initialize) and allocates the memory it needs (Zoltan_Create). Through calls to Zoltan_Set_Fn, the application passes pointers to its callback functions to Zoltan. It also selects a partitioning method (Zoltan_LB_Set_Method) and sets appropriate parameters for load balancing(Zoltan_LB_Set_Param). Then, within the main computation loop, the application calls Zoltan_LB_ Balance to compute a new partition of its data. As a first step in Zoltan_LB_Bal- ance, Zoltan must build the data struc- tures needed for the particular parti- tioning method selected. It calls the 90 1521-9615/02/$17.00 © 2002 COMPUTING IN SCIENCE & ENGINEERING ZOLTAN DATA MANAGEMENT SERVICES FOR P ARALLEL DYNAMIC APPLICATIONS By Karen Devine, Erik Boman, Robert Heaphy, Bruce Hendrickson, and Courtenay Vaughan T HE ZOLTAN LIBRARY IS A COLLECTION OF DATA MANAGE- MENT SERVICES FOR PARALLEL, UNSTRUCTURED, ADAP- TIVE, AND DYNAMIC APPLICATIONS THAT IS AVAILABLE AS OPEN- SOURCE SOFTWARE FROM WWW.CS.SANDIA.GOV/ZOLTAN. IT Editor: Paul F. Dubois, paul@pfdubois.com PROGRAMMING S CIENTIFIC P ROGRAMMING Zoltan Details The Zoltan library’s toolkit includes parallel partitioning algorithms, data migration tools, distributed directo- ries, and unstructured communica- tion and memory management packages. It has many key features for developers: • Its source code is freely available at www.cs.sandia.gov/Zoltan. • It’s callable from C, C++, and F90 (but is implemented in C). • It uses MPI communication.