An Experimental Study to Evaluate a SPL Architecture Regression Testing Approach Paulo Anselmo da Mota Silveira Neto 1,3 , Ivan do Carmo Machado 2,3 , Yguarata Cerqueira Cavalcanti 1,3 Eduardo Santana de Almeida 2,3 , Vinicius Cardoso Garcia 1,3 , Silvio Romero de Lemos Meira 1,3 1 Federal University of Pernambuco (UFPE) – Recife, Pernambuco, Brazil 2 Federal University of Bahia (UFBA) – Salvador, Bahia, Brazil 3 Reuse in Software Engineering (RiSE) {pamsn, vcg, ycc,srlm}@cin.ufpe.br {ivanmachado, esa}@dcc.ufba.br Abstract—In Software Product Line Engineering, where products are derived from a common platform, the reference architecture should be considered the main asset. In order to maintain its correctness and reliability after modifications, a regression testing approach based on architecture specification and code was developed. In this paper, we evaluate it in two different scenarios, the corrective scenario, in which it was performed after a corrective change in the code, and in the progressive scenario, executed after a evolution or enhancement, when some specification changes. The results showed that the progressive scenario was less costly than the corrective. Our evaluation also highlights the importance of a code-based technique to select a set of efficient and effective test cases. Keywords-Software Testing; Software Product Lines; Soft- ware Engineering; Regression Testing; I. I NTRODUCTION Software architectures are becoming the central part dur- ing the development of quality systems [1], being the first model and base to guide the implementation [2] providing a promising way to deal with large and intensive sys- tems [3]. Moreover, it evolves over time to meet customer needs, environment changes, improvements or corrective modifications. To be confident that these modifications are in conformance with the architecture specification, do not introduce unexpected errors, and the new features work as expected, regression tests are performed [4]. There are some approaches [2], tools [5] and method- ologies in the literature, delving into the issues on how to perform regression testing on software architectures, and the most suitable ways to compare two different code versions. From an industry point of view, with the growing Software Product Line Engineering (SPL) adoption, more efficient and effective testing methods and techniques are needed [6], since the available ones make testing a very difficult, expensive and challenging process [7]. More specifically, industry seeks effective SPL regression testing techniques to reduce the amount of retesting [8]. Even with the importance given to regression testing techniques, there are few reports and empirical studies available in the SPL area, and in general, they have been conducted as informal case studies without sufficient at- tempt to scientific rigor from empirical research methods [6]. Thus, additional experimental studies are necessary to provide more evidences about the use of regression testing techniques for SPL development. In this context, this study describes the evaluation of a regression testing approach [9] aimed to reduce the testing effort, by reusing test cases, execution results, as well as to select and prioritize a more effective set of test cases, in terms of performance improvements for fault detection. Taking advantage of SPL architectures similarities, this approach can be applied among product architectures and between the reference and product architectures. II. A REGRESSION TESTING APPROACH FOR SPL ARCHITECTURES According to McGregor [10], regression testing is a technique rather than a testing level. Considering this point of view, regression testing can be performed after any testing level. In our context, it was performed after integration testing, since the purpose of the approach is to verify the integration among modules and components, which compose the SPL architectures. It aims to check if new defects were introduced into previous tested architecture and if it continues working properly. To be confident that the architecture is working properly, its specification can be used as a test oracle to identify when the tests pass or fail. The required inputs for the SPL regression testing ap- proach [9] are both SPL architecture versions, the previous executed integration tests and scripts. The architectural specifications and views (structural and behavioral), and the feature model, product map and feature dependency diagram can be useful assets from which architecture-related informa- tion can be extracted, serving as a guide to identify elements that need to be retested. By using the architecture structural view, the relationship between classes and components are easily specified and identified. This is the process of building the integration tests to be used in the regression approach. The feature model and feature dependency diagram are 608 IEEE IRI 2012, August 8-10, 2012, Las Vegas, Nevada, USA 978-1-4673-2284-3/12/$31.00 ©2012 IEEE