Comparing and Improving Current Packet Capturing Solutions based on Commodity Hardware Lothar Braun, Alexander Didebulidze, Nils Kammenhuber, Georg Carle Technische Universität München Institute for Informatics Chair for Network Architectures and Services {braun,didebuli,kammenhuber,carle}@net.in.tum.de ABSTRACT Capturing network traffic with commodity hardware has be- come a feasible task: Advances in hardware as well as soft- ware have boosted off-the-shelf hardware to performance lev- els that some years ago were the domain of expensive special- purpose hardware. However, the capturing hardware still needs to be driven by a well-performing software stack in order to minimise or avoid packet loss. Improving the cap- turing stack of Linux and FreeBSD has been an extensively covered research topic in the past years. Although the ma- jority of the proposed enhancements have been backed by evaluations, these have mostly been conducted on different hardware platforms and software versions, which renders a comparative assessment of the various approaches difficult, if not impossible. This paper summarises and evaluates the performance of current packet capturing solutions based on commodity hardware. We identify bottlenecks and pitfalls within the capturing stack of FreeBSD and Linux, and give explana- tions for the observed effects. Based on our experiments, we provide guidelines for users on how to configure their captur- ing systems for optimal performance and we also give hints on debugging bad performance. Furthermore, we propose improvements to the operating system’s capturing processes that reduce packet loss, and evaluate their impact on cap- turing performance. Categories and Subject Descriptors C.2.3 [Network Operation]: Network Monitoring General Terms Measurement, Performance 1. INTRODUCTION Packet capture is an essential part of most network mon- itoring and analysing systems. A few years ago, using spe- cialised hardware—e.g., network monitoring cards manufac- Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. To copy otherwise, to republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. IMC’10, November 1–3, 2010, Melbourne, Australia. Copyright 2010 ACM 978-1-4503-0057-5/10/11 ...$10.00. tured by Endace [1]—was mandatory for capturing Gigabit or Multi-gigabit network traffic, if little or no packet loss was a requirement. With recent development progresses in bus systems, multi-core CPUs and commodity network cards, nowadays off-the-shelf hardware can be used to capture net- work traffic at near wire-speed with little or no packet loss in 1 GE networks, too [2, 3]. People are even building mon- itoring devices based on commodity hardware that can be used to capture traffic in 10 GE networks [4, 5] However, this is not an easy task, since it requires careful configuration and optimization of the hardware and software components involved—even the best hardware will suffer packet loss if its driving software stack is not able to handle the huge amount of network packets. Several subsystems including the network card driver, the capturing stack of the operating system and the monitoring application are in- volved in packet processing. If only one of these subsystems faces performance problems, packet loss will occur, and the whole process of packet capturing will yield bad results. Previous work analysed [2, 4, 6] and improved [5, 7, 8] packet capturing solutions. Comparing these work is quite difficult because the evaluations have been performed on dif- ferent hardware platforms and with different software ver- sions. In addition, operating systems like Linux and FreeBSD are subject to constant changes and improvements. Compar- isons that have been performed years ago therefore might today not be valid any longer. In fact, when we started our capturing experiments, we were not able to reproduce the re- sults presented in several papers. When we dug deeper into the operating systems’ capturing processes, we found that some of our results can be explained by improved drivers and general operating systems improvements. Other differ- ences can be explained by the type of traffic we analysed and by the way our capturing software works on the appli- cation layer. While it is not a problem to find comparisons that state the superiority of a specific capturing solution, we had difficulties to find statements on why one solution is superior to another solution. Information about this topic is scattered throughout different papers and web pages. Worse yet, some information proved to be highly inconsistent, espe- cially when from Web sources outside academia. We there- fore encountered many difficulties when debugging the per- formance problems we ran into. This paper tries to fill the gap that we needed to step over when we set up our packet capturing environment with Linux and FreeBSD. We evaluate and compare different cap- turing solutions for both operating systems, and try to sum- marise the pitfalls that can lead to bad capturing perfor- 206