Obtaining High Throughput in Networks with Tiny Buffers Neda Beheshti*, Yashar Ganjali†, Ashish Goel‡, Nick McKeown* *Department of Electrical Engineering, Stanford University {nbehesht, nickm}@stanford.edu †Department of Computer Science, University of Toronto yganjali@cs.toronto.edu ‡Department of Management Science and Engineering, Stanford University ashishg@stanford.edu Abstract— In this paper we explore whether a general topology network built up of routers with very small buffers, can maintain high throughput under TCP’s congestion control mechanism. Recent results on buffer sizing challenged the widely used assumption that routers should buffer millions of packets. These new results suggest that when smooth TCP traffic goes through a single tiny buffer of size O(log W ), then close-to-peak throughput can be achieved; W is the maximum window size of TCP flows. In this work, we want to know if a network of many routers can perform well when all buffers in the network are made very small, independent of the structure of the network. This scenario represents a real network where packets go through several buffering stages on their routes. Assuming the ingress TCP traffic to a network is paced, we first prove that all routers can get by with very small buffers, if the network has a tree structure. For networks with general topology, we propose a simple active queue management policy called Bounded Jitter Policy (BJP), and show that under the proposed policy each flow will preserve its smooth pattern across the network. Logarithmic size buffers would therefore be enough in every router of the network. I. I NTRODUCTION Until recently, Internet routers were widely believed to need large buffers. A typical core router today, can buffer millions of packets, and the buffer size grows linearly as the capacity of the network links increases over time. Recent results on buffer sizing challenged this widely used assumption, and suggested that a smooth and well- multiplexed TCP traffic makes the routers achieve high throughput even when the buffers are made very small [1], [2], [4]. It is shown that implementing buffers of size O(log W ) packets suffices to have close-to-peak throughput when the packets in each TCP flow are sufficiently spaced out [1]. Here, W is the maximum congestion window size of the TCP flows going through the buffer. The tiny buffer results [1] assumes that traffic is paced, either by implementing paced TCP at source (source-paced traffic), or as a result of having very slow access links (link-paced traffic). If this paced traffic goes through one buffering point, then tiny buffers are enough to achieve high throughput. In a real packet-switched network, however, packets go through several buffering points on their routes from source to destination. Even a single router in today’s Internet, usually has more than one buffering stage. A combined input output queued (CIOQ) router, for example, has buffers both on ingress and egress ports. Going through many buffers – even if they are not congestion points – can change traffic pattern along the links. In this work, we explore whether we can obtain a core network of very small buffers and yet keep the utilization high on every link. To answer this question, we consider a network of routers where packets go through at most n routers on their paths. We show that if TCP traffic is paced as it enters the network, then the size of every buffer in the network can be reduced to O(log W + log n) without losing more than a small fraction of throughput on any link in the network. The condition of having paced traffic at ingress ports is automatically satisfied when traffic entering the core is aggregated from slow access networks. To study a network of routers, we first consider a tree-structured network, and show that such structure does not add to the burstiness of traffic as packets move forward in the network. More precisely, we show that any arbitrary buffer of such network would have lost no less throughput, had the arrival traffic been directly injected into that buffer (without going through the prior stages of buffering). A router in this network would therefore require buffers as big as what a single isolated router with the same input traffic requires: when the ingress TCP traffic is non-bursty, shrinking buffers of all routers to a logarithmic size maintains high throughput on every link in the network. Next, we consider a network with general topology. In a network with arbitrary traffic matrix and topology, the characteristics of individual flows might change as they get aggregated with other flows on a link, or when they share queues with some cross-traffic. As a result, a smooth traffic at the source, or at ingress ports of the network, may not preserve its advantageous non-bursty attribute as it traverses across the network. Examples in [10] show network settings where delay jitter increases as traffic goes through more hops toward its destination, despite having only smooth (constant-bit-rate or Poisson) traffic generated at sources. Increased burstiness necessitates bigger buffers to absorb the bursts. To eliminate the possibility of having increased burstiness caused by cross traffic, and consequently the need to have bigger buffers, we propose a simple active queue management policy, which we refer to as Bounded Jitter Policy (BJP). We show that by implementing BJP at each router of a general topology network, all packets will experience an almost fixed delay at each router. As a result, the inter-arrival time of packets in a flow remains unchanged as the flow traverses across the network. The packet arrival sequence of an arbitrary flow at any intermediate router will therefore be a delayed copy of its arrival sequence at the ingress port of the network: if the traffic is spaced out at the ingress port, it will remain spaced out and smooth at any node in the network. Under BJP, a fixed delay can be achieved at the expense of some small additional drops compared with a buffer of the same size under FIFO policy. In return for dropping slightly more packets, our proposed policy guarantees that each flow will preserve its characteristic all along its route in the network. Consequently, if the ingress traffic of the network is sufficiently paced, then tiny buffers of logarithmic size will be sufficient to achieve high throughput between any arbitrary pair of ingress-egress ports. II. BACKGROUND: BUFFER SIZING IN A SINGLE ROUTER Internet routers need buffers to hold packets during times of congestion, and to resolve contention. The size of this buffer directly impacts the packet drop rate, as well as the delay of packets going through the router. Historically, the buffer sizes in Internet core routes