Benchmarking the Robustness of Web Services Marco Vieira, Nuno Laranjeiro, Henrique Madeira CISUC, Department of Informatics Engineering University of Coimbra – Portugal mvieira@dei.uc.pt, cnl@student.dei.uc.pt, henrique@dei.uc.pt Abstract This paper proposes an approach for the evaluation of the robustness of web services, which are complex software components that must provide a robust inter- face to the client applications. However, although web services are becoming business-critical components, there is no practical way to assess the robustness of the code or to compare alternative implementations con- cerning robustness. The approach proposed is based on a set of robustness tests (i.e., invalid web services call parameters) that is applied in order to discover both programming and design errors. The web services are classified based on the failures observed during the execution of the tests. The approach is illustrated by evaluating several web services publicly available in the Internet and two different implementations of the web services specified by the standard TPC-App per- formance benchmark. The proposed approach is useful for both web services providers (to assess the robust- ness of their web services code) and consumers (to select the web services that best fit their requirements). 1. Introduction Web services provide a simple interface between a provider and a consumer and are a strategic mean for data exchange and content distribution. In fact, ranging from on-line stores to media corporations, web services are becoming a key component within the organiza- tions information infrastructure. Several studies show a clear predominance of soft- ware faults (i.e., program defects or bugs) [1, 2] as the root cause of computer failures and, given the huge complexity of today’s software, the weight of software faults will tend to increase. Web services are no excep- tion, as they are normally intricate software compo- nents that implement a compound service, in some cases using compositions of several web services [3], which make them even more complex. Interface faults, related to problems in the interac- tion among software components/modules [4], are par- ticularly relevant in an environment based in web ser- vices. In fact, web services must provide a robust inter- face to the client applications even in the presence of invalid inputs, which may occur due to bugs in the client applications, corruptions caused by silent net- work failures, or even security attacks. Robustness testing has become an attractive ap- proach to characterize the behavior of a system in pres- ence of erroneous inputs [5, 6, 7]. Robustness tests stimulate the system under testing in a way that trig- gers internal errors. Systems can be differentiated ac- cording to the number and type of errors uncovered. Although web services are increasingly being used in complex business-critical systems, current develop- ment support tools do not provide any practical way to assess the robustness of web services code or to com- pare alternative solutions. This paper proposes an ap- proach for the evaluation of the robustness of web ser- vices. The approach consists of a set of robustness tests (i.e., invalid web services call parameters) that is ap- plied during execution in order to disclose both pro- gramming and design problems. The robustness of the web services is characterized according to the follow- ing failure modes: Catastrophic, Restart, Abort, Silent, and Hindering (adopted from the CRASH scale [6]). The proposed benchmark is useful for programmers and testers (in both the provider and consumer sides) in four different scenarios: 1) help providers in evaluating and improving the robustness of their web services code before deployment; 2) help consumers to pick the web services that best fit their requirements by com- paring different alternatives; 3) help providers and con- sumers to identify the need for web-service wrappers that perform parameters validation before web-service invocation; and 4) help providers and consumers to choose the best web services for a given composition 1 . It is important to emphasize that in many situations several solutions implementing the same service are available for consumers. In fact, different implementa- tions may be offered by the same or different provid- 1 Composite web services providers are also consumers of the ser- vices used in the composition.