Deploying Hard Real-time Control Software on Chip-multiprocessors Dai N. Bui University of California, Berkeley EECS Department Berkeley, U. S. A. daib@eecs.berkeley.edu Hiren D. Patel University of Waterloo ECE Department Waterloo, Canada hdpatel@uwaterloo.ca Edward A. Lee University of California, Berkeley EECS Department Berkeley, U. S. A. eal@eecs.berkeley.edu Abstract—Deploying real-time control systems software on multiprocessors requires distributing tasks on multiple processing nodes and coordinating their executions using a protocol. One such protocol is the discrete-event (DE) model of computation. In this paper, we investigate distributed discrete-event (DE) with null-message protocol (NMP) on a multicore system for real-time control software. We illustrate analytically and experimentally that even with the null-message deadlock avoidance scheme in the protocol, the system can deadlock due to inter-core message dependencies. We identify two central reasons for such deadlocks: 1) the lack of an upper-bound on packet transmission rates and processing capability, and 2) an unknown upper-bound on the communication network delay. To address these, we propose using architectural features such as timing control and real-time network-on-chips to prevent such message-dependent deadlocks. We employ these architectural techniques in conjunction with a distributed DE strategy called PTIDES for an illustrative car wash station example and later follow it with a more realistic tunnelling ball device application. Index Terms—Real-time software, Chip-multiprocessors, Discrete-Event. I. I NTRODUCTION The use of multiprocessors for real-time control systems requires distributing the real-time software on the various processing nodes, and coordinating their execution and com- munication with a protocol. Two common protocols are time- triggered and event-triggered. With either of these protocols, it is critical to ensure that the system is deadlock free. In this paper, we focus on the design of reliable real-time systems on chip-multiprocessors (CMPs) using the event-triggered proto- col; in particular, the discrete-event (DE). However, we do not consider a monolithic event queue because of its inability to exploit parallelism, and its susceptibility to being a single point of failure. Alternatively, we concentrate on distributing the discrete-event execution across multiple processing nodes. In doing this, we study the effects of distributing DE with a deadlock avoidance mechanism known as the null-message protocol (NMP) [1], [2]. Specifically, we evaluate a potential message-dependent deadlock [3], [4] problem that arises even when NMP is used. The architecture we use is a CMP with multiple processing nodes connected via a network-on-chip (NoC). A processing node connects to a network interface, which is directly con- nected to the NoC interconnect. Designers of NoCs often as- sume that packets transmitted to a processing node are always consumed immediately. With this assumption, the designer provides guarantee that sent packets are always delivered. This means that once a packet is sent, it will reach the destination within a finite amount of time. Therefore, there is no deadlock or livelock in the network that may cause a packet to never reach its destination. However, in implementations of CMPs, processing nodes (i.e. CPUs) have limited memory and processing resources; therefore, processing nodes cannot always consume packets as soon as they arrive. If too many packets arrive at a processing node during an interval, then they are usually queued up in the network. This results in a blocking effect in the network, which might cause the system to deadlock entirely or partially. This is called message- dependent deadlock [3], [4]. In the case of NMP, each processing node regularly sends null messages to some other processing nodes to update those nodes about the sender’s physical time. It is very possible that when too many null messages (packets) are sent to the same receiving node, if that node is busy doing some task then it cannot process these messages. Therefore, these null messages fill up the input buffer at the network interface of the node. This congestion prevents other non-null messages from being processed quickly as well. This temporary blocking effect is problematic for real-time systems because it might cause the system to miss its real-time deadline simply due to a congestion caused by null messages. Moreover, if the buffer capacity reaches its maximum, then the null messages can result in blocking the entire network; essentially, a deadlock. We show that distributing DE on a CMP using the NMP protocol can result in message-dependent deadlocks. These deadlocks arise from two factors: 1) bursty traffic due to network jitter and 2) mismatch in transmit and receive rates. We identify these factors via an analytical model based on the ideas of network calculus [5] and real-time calculus [6]. Then, we discuss architectural features that can address these factors. In particular, we present an implementation that combines a real-time network-on-chip communication network that elim- inates jitter [7] with a real-time processor architecture called the Precision Timed (PRET) machine [8] with instructions for timing control. We show that by using timing instructions in software we can enforce a bound on the message transmit-