Configuration Knowledge of Software Product Lines: A Comprehensibility Study Elder Cirilo 1 , Ingrid Nunes 1,2 , Alessandro Garcia 1 , Carlos J.P. de Lucena 1 1 Pontifical Catholic University of Rio de Janeiro (PUC-Rio), Rio de Janeiro, Brazil {ecirilo,ionunes,afgarcia,lucena}@inf.puc-rio.br 2 King’s College London, Strand, London, WC2R 2LS, United Kingdom ABSTRACT The configuration knowledge is a key element to the suc- cess of software product lines, as it defines constraints on how product line variability should be composed to derive products. Even though configuration knowledge specifica- tion is a long standing problem in software product line en- gineering, the impact of different specification techniques on comprehensibility has never been studied. This paper presents an empirical study to evaluate and compare three techniques for configuration knowledge specification. Each of them is centered on different means to express the config- uration constraints: annotations, general-purpose modeling, and domain-specific modeling. Our results suggest that: (i) the use of domain-specific abstractions tends to facilitate the comprehension of coarse-grained variability; (ii) the use of general-purpose models imposes certain restrictions on the location and comprehension of the configuration knowl- edge; and (iii) the correct comprehension of configuration constraints is not associated with individual expertise. Categories and Subject Descriptors D.2.11 [Software Engineering]: Software Architectures— Domain-specific architectures ; D.2.8 [Software Engineer- ing]: Metrics General Terms Experimentation Keywords Software Product Lines, Configuration Knowledge, Controlled Experiment 1. INTRODUCTION The adoption of configurable product lines for building families of enterprise systems is increasingly growing. A Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. To copy otherwise, to republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. VariComp ’11 Porto de Galinhas, Pernambuco, BR Copyright 2011 ACM 978-1-4503-0646-1/11/03 ...$10.00. configurable product line [4] is a subclass of software prod- uct lines that is tailored into single products without de- signing or programming new completion code. Systematic variability management as configuration potentially results in significant gains in software development such as reduced time-to-market and costs, but it requires explicitly speci- fying the configuration knowledge. This knowledge defines variability implementation and how it should be composed to derive products. If the activity of localizing and under- standing the configuration knowledge requires a significant effort and costs to be performed, there is no payoff for the upfront investment made to build reusable software assets. Therefore, an easy-to-comprehend specification of all vari- ability implementation and valid compositions is essential to the success of configurable product lines. A particularity of enterprise product lines is the reuse of a large number of frameworks in concert for addressing a di- versity of concerns (e.g., service composition and graphical user interfaces), mainly because they are ready-to-use infras- tructures. As frameworks typically rely on a set of domain- specific abstractions, these become part of mindset of prod- uct line engineers along the software development. However, implementing variability as domain-specific abstractions in- volves making completion choices, some of which cut across heterogeneous languages (e.g., Java and XML). Since the completion code is not found in a single location, it tends to result in variability implementation spread across the prod- uct line architecture. As a consequence, it can lead to a negative impact on the activities of localizing and under- standing the configuration knowledge specification. A diversity of specification techniques were introduced as a way to facilitate the comprehension of the configuration knowledge [1, 2, 3]. Nevertheless, there are limited investi- gations about the implications of the adopted technique on the comprehensibility of framework-based software product lines. In fact, as far as we know, there is no study that analyzes the impact of specifying the configuration knowl- edge using the different available techniques, namely anno- tations, general-purpose modeling and domain-specific mod- eling. Existing research work [5] has analyzed and com- pared the individual techniques in terms of modularity, but they have not observed their level of impact on the configu- ration knowledge comprehensibility measures. In addition, their findings are not applicable to the context of enterprise product lines built with multiple frameworks, which is the scenario we are addressing. We have performed an experimental study to evaluate and compare three tools that support the aforementioned tech-