JOURNAL OF SOFTWARE MAINTENANCE AND EVOLUTION: RESEARCH AND PRACTICE J. Softw. Maint. Evol.: Res. Pract. 2003; 15:41–59 (DOI: 10.1002/smr.265) Research Test processes in software product evolution—a qualitative survey on the state of practice Per Runeson ∗,† , Carina Andersson and Martin H¨ ost Software Engineering Research Group, Department of Communication Systems, Lund University, Box 118, SE-221 00 Lund, Sweden SUMMARY In order to understand the state of test process practices in the software industry, we have conducted a qualitative survey, covering software development departments at 11 companies in Sweden of different sizes and application domains. The companies develop products in an evolutionary manner, which means either new versions are released regularly, or new product variants under new names are released. The survey was conducted through workshop and interview sessions, loosely guided by a questionnaire scheme. The main conclusions of the survey are that the documented development process is emphasized by larger organizations as a key asset, while smaller organizations tend to lean more on experienced people. Further, product evolution is performed primarily as new product variants for embedded systems, and as new versions for packaged software. The development is structured using incremental development or a daily build approach; increments are used among more process-focused organizations, and daily build is more frequently utilized in less process-focused organizations. Test automation is performed using scripts for products with focus on functionality, and recorded data for products with focus on non-functional properties. Test automation is an issue which most organizations want to improve; handling the legacy parts of the product and related documentation presents a common problem in improvement efforts for product evolution. Copyright c  2003 John Wiley & Sons, Ltd. KEY WORDS: qualitative survey; verification and validation; industry practice; interviews; evolution 1. INTRODUCTION Verification and validation activities take a substantial share of project budgets. Early rules of thumb devoted 50% of the time schedule to testing [1], and no great breakthroughs seem to have changed this dramatically. An earlier survey, with a focus on lead-time consumption, concludes that there is ∗ Correspondence to: Per Runeson, Software Engineering Research Group, Department of Communication Systems, Lund University, Box 118, SE-221 00 Lund, Sweden. † E-mail: per.runeson@telecom.lth.se Received 30 June 2002 Copyright c  2003 John Wiley & Sons, Ltd. Revised 25 September 2002