Space–Parallel Network Simulations using Ghosts * George F. Riley 1 Talal M. Jaafar 1 Richard M. Fujimoto 2 Mostafa H. Ammar 2 1 Department of Electrical and Computer Engineering Georgia Institute of Technology Atlanta, GA 30332-0250 {riley,jaafar}@ece.gatech.edu 2 College of Computing Georgia Institute of Technology Atlanta, GA 30332-0280 {fujimoto,ammar}@cc.gatech.edu March 4, 2004 Abstract We discuss an approach for creating a federated network sim- ulation that eases the burdens on the simulator user that typi- cally arise from more traditional methods for defining space– parallel simulations. Previous approaches have difficulties that arise from the need for global topology knowledge when for- warding simulated packets between the federates. In all but the simplest cases, proper packet forwarding decisions between fed- erates requires routing tables of size O(mn) (m is the number of nodes modeled in a particular simulator instance, and n is the total number of network nodes in the entire topology) in order to determine how packets should be routed between federates. Further, the benefits of the well–known NIx-Vector routing ap- proach cannot be fully achieved without global knowledge of the overall topology. We seek to overcome these difficulties by utilizing a topology partitioning methodology that uses Ghost Nodes. A ghost node is a simulator object in a federate that represents a simulated network node that is spatially assigned to some other federate, and thus that other federate is responsi- ble for maintaining all state associated with the node. However, ghost nodes do retain topology connectivity information with other nodes, allowing all federate in a space–parallel simulation to obtain a global picture of the network topology. We show with experimental results that the memory overhead associated with the ghosts is minimal relative to the overall memory footprint of the simulation. 1 Introduction One approach to creating simulation models for large–scale topologies in network simulations is to use a space–parallel partitioning methodology, coupled with distributed simulation methods. In a space–parallel network simulation, the model for * This work is supported in part by NSF under contract numbers ANI- 9977544, ANI-0136969, ANI-0240477, ECS-0225417, and DARPA un- der contract number N66002-00-1-8934. the entire simulated network is divided logically into k sub– models, where k is the number of federates in the distributed simulation. With this approach, each federate is responsible for approximately 1/k th of the entire topology model, and instan- tiates simulation objects to represent its own portion of the net- work elements in the complete topology. Since a given federate has no responsibility for the remaining (k - 1)/k portion of net- work elements, no simulation objects are created and thus the federate has no knowledge about the remaining topology. This approach is fairly easy to implement, and is the method used by existing space–parallel distributed network simulators such as Parallel/Distributed ns (pdns) and the Georgia Tech Network Simulator (GTNetS) [1, 2]. Further, this method has very good scalability, since each federate need only be concerned with its own network elements, and thus only allocates simulator mem- ory for a fraction of the entire set of network elements. However, as we shall show this approach introduces a number of difficul- ties that must be addressed in order to insure correct packet for- warding between the federates. Our solution to these difficulties is to introduce the notion of a Ghost Node. A ghost node is a simulator object that acts as a placeholder for nodes that are assigned to other federates. The ghost node object has none of the complex and memory intensive state needed for real nodes (such as queues, routing tables, port maps, and applications). Rather, a ghost node sim- ply contains topology connectivity information about links and neighbors. Thus, using ghosts, a federate is afforded a global picture of the simulated topology, without the memory overhead of maintaining unneeded state for the ghosts. The remainder of this paper is organized as follows. Section 2 discusses the space–parallel approach for distributed network simulation, and shows some of the difficulties associated with these traditional approaches. Section 3 gives the basic design of our Georgia Tech Network Simulator with emphasis on the ghost node implementation. Section 4 discusses related work. Section 5 presents memory usage statistics comparing our ghost node approach to more traditional routing table approaches. Fi- nally, section 6 describes conclusions from this work.