Congestion Avoidance with Incremental Filter Aggregation in Content-Based Routing Networks Mingwen Chen 1 , Songlin Hu 1 , Vinod Muthusamy 2 , Hans-Arno Jacobsen 3 1 Chinese Academy of Sciences, 2 IBM T.J. Watson Research Center, 3 University of Toronto Abstract—The subscription covering optimization, whereby a general subscription quenches the forwarding of more specific ones, is a common technique to reduce network traffic and rout- ing state in content-based routing networks. Such optimizations, however, leave the system vulnerable to unsubscriptions that trigger the immediate forwarding of all the subscriptions they had previously quenched. These subscription bursts can severely congest the network, and destabilize the system. This paper presents techniques to retain much of the benefits of subscription covering while avoiding bursty subscription traffic. Heuristics are used to estimate the similarity among subscriptions, and a distributed algorithm determines the portions of a subscription propagation tree that should be preserved. Evaluations show that these mechanisms avoid subscription bursts while maintaining relatively compact routing tables. I. I NTRODUCTION Route summarization is a common technique used by routers to reduce routing table size and improve routing performance. In IP-networks, addresses are allocated in a hierarchical manner, and it is natural for routers to aggregate a set of addresses based on this hierarchy. For example, all nodes in the 10.0.0.0/8 subnet can be represented by a single routing table entry for the entire subnet [1]. This summarization is also popular in distributed content- based publish/subscribe systems. For example, subscription covering is a common filter summarization technique in content-based publish/subscribe systems, which exploits simi- larity among subscriptions and takes advantage of the property that a router need not index a specific subscription when there is a more general one present [2], [3]. Consider a content- based router with a general subscription S and a set of more specific subscriptions that are covered by S. The covering optimization will only forward S and avoid forwarding the covered subscriptions. In this way, the covering optimization has many benefits including smaller routing tables, fewer subscription propagations, and faster routing decisions [3], [4]. Unlike IP routers, content-based routers cannot rely on a static hierarchical subnet structure. Instead, node “addresses”, that is, their content-based subscription, may change and therefore the subscription aggregations need to be updated to reflect the current set of subscriptions in the system. This is a problem confronted in our previous work [5]. Consider a content-based router with a general subscription S. When S is to be removed, perhaps because the client is not interested This work is supported by the National Natural Science Foundation of China under Grant No.61070027, 61020106002 and 61161160566. Between 2008 to 2010, this work was supported by an NSERC Discovery Grant. in S any more, it becomes necessary to replace S with the more precise, but also more numerous, covered subscriptions. In this paper, we use S to denote subscriptions that are triggered by S, Since the subscriptions in S are no longer covered by any subscription, all the subscriptions in S need to be forwarded. To maintain correctness of the routing tables, all the subscriptions in S are forwarded immediately before removing S. As it is more costly to process subscriptions than other messages in covering-based routing [6], a large set of subscription messages can impose severe congestion in the network, overwhelming the brokers that must process this burst of subscription traffic, and destabilize the system. 0 5 10 15 20 25 0 40 80 120 160 Queued messages (x1000) Time (min) Fig. 1. Subscription bursts triggered by un- subscriptions can result in severe congestion that persists for a long time. (Setup detailed in Sec. V.) To illustrate the potential severity of this problem, Fig. 1 quantifies how broker queue lengths are affected by the burst of subscriptions that result from removing a small number of subscriptions. Here, the system only stabilizes after several hours. What’s more, in systems with limited queue sizes, the burst of subscriptions will result in message loss. The problem addressed in this paper may arise in any scenario with a high degree of subscription covering where some subscriptions are cancelled or modified. Here are some examples: (1) In a distributed workflow management system, each workflow activity issues subscriptions that reflect its narrow interest in the events that trigger it [7]–[10]. As well, an administrator wanting to monitor the entire system would issue a broad subscription S that covers many of the existing ones. When monitoring stops, the removal of the associated subscription S will result in a large and instantaneous burst of many subscriptions heretofore covered by S. (2) In a content delivery system [11]–[13], users may express both broad and narrow interests and share similar interests [14]. As a matter of fact, a general subscription might cover thousands of more specific subscriptions. For example, in a mobile content delivery system, drivers may subscribe to updates about traffic in their vicinity, while a traffic reporter needs to subscribe to all road conditions. When the reporter leaves or disconnects from the system, there is a risk of subscription bursts [15]. (3) In a financial system, analysts subscribe to updates about