The Effect of the Router Arbitration Policy on the Scalability of ServerNet Topologies V.Shurbanov , D.R.Avresky , and R.Horst Network Computing Research Laboratory Tandem Computers Incorporated Dept. of ECE, Boston University 19333 Vallco Parkway 15 St. Mary's St., Boston, MA 02215 Cupertino, CA 95014 Phone:(617) 353-9850 Phone: (408) 285-1816 Fax: (617) 353-6440 Fax: (408) 285-1819 E-mail: avresky,vash @bu.edu E-mail: horst bob@tandem.com Abstract In this paper we extend a previously introduced method for optimizing the arbitration policy employed by Server- Net routers and we evaluate the method's effect on scala- bility. The ServerNet System Area Network (SAN), de- veloped by Tandem Computers Inc., is a wormhole-routed, packet-switched, point-to-point network, with special atten- tion paid to reducing latency and to assuring reliability. The ServerNet SAN uses multiple high-speed, low-cost routers to rapidly switch data directly between data sources and destinations. The ServerNet router arbitration algorithm has the abil- ity to assign different priorities to different links thus con- trolling the distribution of available bandwidth among channels. We propose an analytical method that allows us to determine the correct priorities for channels so as to achieve fair arbitration thus increasing throughput and scalability, minimizing latency, and eliminating tree satura- tion. In our study we use data generated by a simulation tool to validate the proposed analytical model for determining the weights of router ports and to evaluate the resulting per- formance improvements. 1 Introduction Multistage networks are finding increasing use in build- ing clusters of workstations and PCs. The networks that support such connectivity must provide high bandwidth, low latency, scalability, low cost, high level of usability, and reliability [4]. A key parameter that influences the network performance characteristics such as maximum la- tency, throughput, scalability, and tree saturation, is the ar- bitration policy implemented in routers. The ServerNet System Area Network (SAN) provides high-speed communications from processor to processor, processor to I/O device, or I/O device to other I/O devices. The development of the ServerNet SAN [3] has prompted the exploration of new topologies in search for better ways of constructing networks to optimize performance, while avoiding the possibility of deadlock. Several ServerNet SAN topologies based on have been studied through sim- ulation in an attempt to estimate their performance char- acteristics including maximum latency, scalability, and tree saturation effects. The weights (i.e., increment values) assigned to router ports have a profound influence on the performance of the network and in particular on the fairness of arbitration (i.e. QoS) because these weights determine the fraction of the channel bandwidth that will be provided to the device(s) connected to a port. We have developed a method that al- lows us to determine the amount of traffic arriving at router port and competing for router port under a particular traffic pattern, and on the basis of this to calculate the port weights so as to provide a fair arbitration scheme i.e., each device should receive bandwidth proportional to its require- ments as compared to the requirements of its competitors. In this paper we extend the initial 16-CPU model and apply it to the 128-CPU Thin and Fat Fractahedron Servet- Net topologies. We also study and evaluate the effect of the improved arbitration on the scalability of these topologies. 2 ServerNet basics The ServerNet SAN is a wormhole-routed, packet- switched, point-to-point network, with special attention paid to reducing latency and to assuring reliability [3]. It uses multiple high-speed, low-cost routers to rapidly switch data directly between data sources and destinations.