Postprint of article in the Journal of Systems and Software 84 (4): 603–619 (2011) XML-manipulating test case prioritization for XML-manipulating services * † Lijun Mei a , W.K. Chan b ‡ , T.H. Tse a , Robert G. Merkel c a Department of Computer Science, The University of Hong Kong, Pokfulam, Hong Kong b Department of Computer Science, City University of Hong Kong, Tat Chee Avenue, Hong Kong c Faculty of Information Technology, Monash University, Clayton, Victoria, Australia A B S T R A C T A web service may evolve autonomously, making peer web services in the same service composition uncertain as to whether the evolved behaviors are compatible with its original collaborative agreement. Although peer services may wish to conduct regression testing to verify the agreed collaboration, the source code of the former service may be inaccessible to them. Owing to the black-box nature of peer services, traditional code-based approaches to regression testing are inapplica- ble. In addition, traditional techniques assume that a regression test suite for verifying a web service is available. The location to store a regression test suite is also a problem. On the other hand, we note that the rich interface specifications of a web service provide peer services with a means to formulate black-box testing strategies. In this paper, we provide a strategy for black-box service-oriented testing. We also formulate new test case prioritization strategies using tags embedded in XML messages to reorder regression test cases, and reveal how the test cases use the interface specifications of web services. We experimentally evaluate the effectiveness of these black-box strategies in revealing regression faults in modified WS-BPEL programs. The results show that the new techniques can have a high chance of outperforming random ordering. Moreover, our experiment shows that prioritizing test cases based on WSDL tag coverage can achieve a smaller variance than that based on the number of tags in XML messages in regression test cases, even though their overall fault detection rates are similar. Keywords: Test case prioritization; Black-box regression testing; WS-BPEL; Service testing; Service-oriented testing 1. Introduction The testing and analysis of web services have posed new foundational and practical challenges, such as the non- observability problem (Canfora and Di Penta, 2006; Mei et al., 2009d), the extensive presence of non-executable artifacts within and among web services (Mei et al., 2008a, 2009b), safeguards against malicious messages from external parties (Xu et al., 2005; Martin et al., 2007), ultra-late binding (Bartolini et al., 2008), and cross-organizational issues (Ye et al., 2009). Researchers have proposed diverse techniques to address the test case selection problem (Martin et al., 2007), the test adequacy problem (Mei et al., 2008b, 2009b), the test oracle problem (Tsai et al., 2005a; Chan et al., 2007), and the test case prioritization problem (Hou et al., 2008; Mei et al., 2009c, 2009d). Regression testing is the de facto activity to address the testing problems caused by software evolution (Onoma et al., 1998). It aims to detect software faults by retesting modified software versions. However, many existing re- gression testing techniques (such as Harrold et al., 1993; Kim and Porter, 2002; Mei et al., 2009c; Rothermel et al., 2001) assume that the source code is available for monitoring (Mei et al., 2009d), and use the coverage information of executable artifacts (such as statement coverage achieved by individual test cases) to conduct regression testing. Nonetheless, the coverage information on an external service may not be visible to the service composition that utilizes this service. Moreover, even * © 2011 Elsevier Inc. This material is presented to ensure timely dissemi- nation of scholarly and technical work. Personal use of this material is permitted. Copyright and all rights therein are retained by authors or by other copyright holders. All persons copying this information are expected to adhere to the terms and constraints invoked by each author’s copyright. In most cases, these works may not be reposted without the explicit permission of the copyright holder. Permission to reprint / republish this material for advertising or promotional purposes or for creating new collective works for resale or redistribution to servers or lists, or to reuse any copyrighted component of this work in other works must be obtained from Elsevier Inc. † This research is supported in part by the General Research Fund of the Research Grant Council of Hong Kong (project no. 717308), a strategic research grant of City University of Hong Kong (project no. 7002464), and a discovery grant of the Australian Research Council (project no. DP0984760). A preliminary version of this paper was presented in QSIC 2009 (Mei et al., 2009c). ‡ Corresponding author. Tel.: +852 3442 9684. E-mail addresses: ljmei@cs.hku.hk (L. Mei), wkchan@cs.cityu.edu.hk (W.K. Chan), thtse@cs.hku.hk (T.H. Tse), robert.merkel@benambra.org (R.G. Merkel). HKU CS Tech Report TR-2010-11