AONBench: A Methodology for Benchmarking XML Based Service Oriented Applications Abdul Waheed Cisco Systems, Inc., 170 W. Tasman Drive, San Jose, CA 94035, USA Email: abwaheed@cisco.com Jianxun Jason Ding Cisco Systems, Inc., 170 W. Tasman Drive, San Jose, CA 94035, USA Email: jiding@cisco.com Abstract—Service Oriented Architectures (SOA) and applications increasingly rely on network infrastructure instead of back-end servers. Cisco Systems Application Oriented Networking (AON) initiative exemplifies this trend. Benchmarking such infrastructure and their services is expected to play an important role in the networking industry. We present AONBench specifications and methodology to benchmark networked XML application servers and appliances. AONBench is not a benchmarking tool. It is a specification and methodology for performance measurements, which leverages from existing XML microbenchmarks and uses HTTP for end-to-end communication. We implement AONBench specifications for end-to-end performance measurements through public domain HTTP load generation tool ApacheBench and Apache web server. We present three case studies of using AONBench for architecting real application oriented networking products. Index Terms—SOA, XML, Application Oriented Networking, Benchmarking. I. INTRODUCTION Traditional network infrastructure is based on providing simple packet switching, forwarding, queuing, and routing services, while complex message-oriented tasks are left to the end hosts. This task partitioning was logical when network infrastructure had limited processing capabilities. With continuing exponential growth in processor performance, this partitioning is becoming increasingly irrelevant. Cisco Systems recently launched its Application Oriented Networks (AON) products that integrate message processing capabilities within the network infrastructure. AON modules can be added to specific switches and routers to provide application-oriented services in addition to default packet switching and forwarding functions. A service oriented architectures with AON can offload an application server endpoint by providing network infrastructure, security, and application-specific integration services. Currently, many application servers have to provide network infrastructure services in addition to supporting their target applications. These services include protocol processing, load balancing, security and firewalls, content caching and delivery, and integrating heterogeneous standards. Cisco AON products have pushed these services to network infrastructure and are available as switch and router embedded modules as well as appliance incarnations. We shall refer to any one of these three incarnations as an Application Oriented Network Infrastructure and Services EnableD (AONISED) node for the sake of consistency. XML is becoming ubiquitous for web services applications. AONISED nodes use XML for message processing in the same manner that IP is used by routers for packet processing. However, in contrast to packet, message processing offers different types of challenges. Unlike IP packets, application messages are typically stateful and belong to specific flows. An AONISED node has to deal with message content instead of merely processing the protocol headers in individual packets. For instance, parsing an XML message for pattern matching is more expensive than packet header manipulation. In addition, security operations on XML message content are inherently compute-intensive. All of these challenges point to the criticality of AONISED node performance. Networked services providers and businesses using their services need a consistent performance measurement methodology to benchmark a growing number of AONISED products. There are multiple benchmarks available for web services such as caching, proxying, content-delivery, and storage networking using HTTP and HTTPS. Similarly, multiple XML microbenchmarks are also available that target specific functions such as parsing, Document Oriented Models (DOM) construction, pattern matching, transformation, validation, and security. A benchmark for an AONISED node needs to merge the functions of a web services benchmark and an XML functions microbenchmark. However, due to AONISED node specific nature of XML interfaces, such a merger will not extend to any other product having a different interface. There are three requirements that an AONISED node benchmark is expected to meet: 46 JOURNAL OF NETWORKS, VOL. 2, NO. 5, SEPTEMBER 2007 © 2007 ACADEMY PUBLISHER