Vol.7 (2017) No. 4-2 ISSN: 2088-5334 Evaluation of Software Product Line Test Case Prioritization Technique Muhammad Sahak # , Shahliza Abd Halim #,* , Dayang Norhayati Abang Jawawi #,* , Mohd Adham Isa #,* # Software Engineering Research Group, Universiti Teknologi Malaysia, Skudai 81310 Johor Malaysia * Software Engineering Department, Universiti Teknologi Malaysia, Skudai 81310 Johor Malaysia E-mail: shahliza@utm.my Abstract— Software product line (SPL) engineering paradigm is commonly used to handle commonalities and variabilities of business applications to satisfy the specific needs or goal of a particular market. However, due to time and space complexities, testing all products is not feasible, and SPL testing is proven to be difficult due to a combinatorial explosion of the number of products to be considered. Combinatorial interaction testing (CIT) is suggested to reduce the size of test suites to overcome budget limitations and deadlines. CIT is conducted to fulfill certain quality attributes. This method can be further improvised through the prioritization of list configuration generated from CIT to gain better results in terms of efficiency and scalability, However, to the best of our knowledge, not much research has been done to evaluate existing Test Case Prioritization (TCP) techniques in SPL. This paper provides a survey of existing works on test case prioritization technique. This study provides classification and compares the best technique, trends, gaps and proposed frameworks based on the literature. The evaluation and discussion are using Normative Information Model-based Systems Analysis and Design (NIMSAD) on aspects that include context, content, and validation. The discussion highlights the lack of technique for scalability issue in SPL with most of the work is on academia setting but not on industrial practices. Keywords— NIMSAD; software product lines; regression testing I. INTRODUCTION Software Product Line (SPL) is founded on the concept of reusability of products from the same family which is systematically being reused either as common assets or only shared by a subset of the family [1]. Many software organizations changed their development process from a single system to SPL to take advantage of the reduction in time, cost, and effort to market while significantly increase the quality of derived products. Among the common quality assurance methods in SPL is SPL testing. The difference between testing a single system and SPL testing is, in a single system, only one product is tested at a time whereas SPL testing tests a number of products at one time which contributes to the need for systematic testing process due to the commonality and variability of features. The testing process becomes more complicated when the number of configuration grows exponentially with the number of features or known as a combinatorial explosion. In addition, computing these large number of products in the presence of constraints proves to be a tough issue and scalability is considered as an open research area in SPL [3]. Thus, it prompts the need for a new method to overcome these challenges. Among the promising methods is a regression testing method which is able to reduce the number of test artifacts in a single system through minimization, selection, and prioritization. This method has been adapted into SPL in the works of [1], [2], [3], [6]. Test Case Prioritization (TCP) is known as the best regression technique that can be used to overcome these issues by rearranging the test cases which cover a lot of changed elements of a product variant in order to achieve the desired criteria. A typical TCP phase in SPL is shown in Fig. 1. This study divides the focus of existing works into two quality attributes which are efficiency and scalability. In terms of efficiency, software tester can benefit the most from prioritization where only a few most important test cases from the test suite will be selected thus helps in increasing the interaction coverage between SPL product under test as fast as possible. Whereas, for scalability issue, the relaxation of the T-wise criterion by using TCP helps contribute to lower testing effort and decent coverage. This is because, in SPL testing, high T-wise criterion requires a lot of budgets. Contributions of this paper are as the following: (i) provide a classification scheme for our own 1601 brought to you by CORE View metadata, citation and similar papers at core.ac.uk provided by International Journal on Advanced Science, Engineering and Information Technology