Security Vulnerability in Processor-Interconnect Router Design WonJun Song, John Kim KAIST Daejeon, Korea {wjsong,jjk12}@cs.kaist.ac.kr Jae W. Lee SungKyunKwan University Suwon, Korea jaewlee@skku.edu Dennis Abts Google Inc. Madison, WI dabts@google.com Abstract Servers that consist of multiple nodes and sockets are in- terconnected together with a high-bandwidth, low latency processor interconnect network, such as Intel QPI or AMD Hypertransport technologies. The different nodes exchange packets through routers which communicate with other routers. A key component of a router is the routing table which de- termines which output port an arriving packet should be forwarded through. However, because of the flexibility (or programmability) of the routing tables, we show that it can result in security vulnerability. We describe the procedures for how the routing tables in a processor-interconnect router can be modified. Based on these modifications, we propose new system attacks in a server, which include both perfor- mance attacks by degrading the latency and/or the band- width of the processor interconnect as well as a livelock at- tack that hangs the system. We implement these system on an 8-node AMD server and show how performance can be significantly degraded. Based on this vulnerability, we propose alternative solutions that provide various trade-off in terms of flexibility and cost while minimizing the routing table security vulnerability. Categories and Subject Descriptors C.2.0 [Computer-Communication Networks]: General– security and protection; C.2.1 [Network Architecture and Design]: network communications, network topology General Terms Security Keywords processor-interconnect; routing table; router; vulnerability Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full cita- tion on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or re- publish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from permissions@acm.org. CCS’14, November 3–7, 2014, Scottsdale, Arizona, USA. Copyright 2014 ACM 978-1-4503-2957-6/14/11 ...$15.00. http://dx.doi.org/10.1145/2660267.2660290. 1. INTRODUCTION Interconnection networks can be found in many differ- ent domains, including on-chip network for multicore ar- chitecture as well as off-chip networks for large-scale sys- tem such as supercomputers [8]. The interconnection net- work is a critical component of modern systems as com- municating or moving data between compute nodes are be- coming more important in determining overall system per- formance and cost. With multisocket servers being com- monly used in high-performance computing and datacenters, we focus on the processor-interconnect or the interconnec- tion networks found in these modern servers with multi- ple sockets and nodes. Each socket can consist of multiple nodes with multi-chip modules and each node can consist of multiple cores. These nodes are interconnected with a processor-interconnect and some commonly used processor- interconnects include Intel QPI [29] and AMD Hypertrans- port [2]. In this work, we focus on the processor-interconnect found in these multi-socket (multi-node) SMP (symmetric multiprocessing) systems and the security vulnerability in the routing tables found within the processor-interconnect router microarchitecture. The basic building block of any interconnection network is the router which receives packets and forwards them to the appropriate output ports. As a result, one of the first steps for an arriving packet is determining the output port and is done by some routing computation logic within the router. To provide flexibility, a routing table is commonly used for the routing computation. The routing table struc- ture is a lookup table, with the lookup done often based on the packet’s destination and the entries within the routing table specifying the output channel 1 that needs to be used to route the packet. Although the routing table provides flexibility, we show in this work that such flexibility can result in security vulnerability. To the best of our knowl- edge, this is one of the first work to investigate security vulnerability within a processor-interconnect; in particular, the routing table that is located within each router. We first describe the methodology for how to modify the routing table, which includes first understanding the topol- ogy (or the connectivity) of the processor-interconnect. Based on the ability to modify the routing table, we present three different types of attacks. The livelock attack modifies the routing table such that packets circulate in the network and 1 In this work, we use the terminology link and channel in- terchangeably. In addition, the term nodes and routers are also used interchangeably since a communication between “nodes” are done through the routers. 358