Transitioning Manual System Test Suites to Automated Testing: An Industrial Case Study Emil Al´ egroth Software Engineering and Technology Chalmers University Gothenburg, Sweden emil.alegroth@chalmers.se Robert Feldt Software Engineering and Technology Chalmers University Gothenburg, Sweden robert.feldt@chalmers.se Helena H. Olsson Department of Computer Science Malm¨ o University Malm¨ o, Sweden helena.holmstrom.olsson@mah.se Abstract—Visual GUI testing (VGT) is an emerging tech- nique that provides software companies with the capability to automate previously time-consuming, tedious, and fault prone manual system and acceptance tests. Previous work on VGT has shown that the technique is industrially applicable, but has not addressed the real-world applicability of the technique when used by practitioners on industrial grade systems. This paper presents a case study performed during an industrial project with the goal to transition from manual to automated system testing using VGT. Results of the study show that the VGT transition was successful and that VGT could be applied in the industrial context when performed by practitioners but that there were several problems that first had to be solved, e.g. testing of a distributed system, tool volatility. These problems and solutions have been presented together with qualitative, and quantitative, data about the benefits of the technique compared to manual testing, e.g. greatly improved execution speed, feasible transition and maintenance costs, improved bug finding ability. The study thereby provides valuable, and previously missing, contributions about VGT to both practitioners and researchers. Keywords-Visual GUI testing; Test Automation; Test Mainte- nance; Empirical; Industrial case study. I. I NTRODUCTION To date, there are no industrial case studies, from the trenches, that visual GUI testing (VGT) works in industry when used by practitioners, nor data to support the long-term viability of the technique. In our previous work, we have shown that VGT is applicable in industry, even for testing of safety-critical software [1]. However, previous work has been essentially driven by researchers, e.g. they applied VGT techniques, compared the resulting test cases to earlier manual efforts, and then collected feedback and refinements from the industrial practitioners. There is a risk that this type of research does not consider all the complexities and problems seen by practitioners when actually applying a technique in practice. Furthermore, researcher driven studies are often smaller in scale and cannot evaluate longer term effects such as maintenance and refactoring of the test scripts or effects on, and of, changes to the system under test (SUT). Hence, there is still a gap in VGT’s body of knowledge regarding if the technique is applicable when performed by industrial practitioners in a real world development context. In this paper we aim to bridge this gap by presenting an industrial case study from a successful project, driven entirely by industrial practitioners, with the goal to transition into VGT at the company Saab AB, subdivision security and defense solutions (SDS). The company chose VGT because of its ability to automate high system-level test cases, which previous automation techniques, e.g. unit testing [2], [3] and record and replay (R&R) [4]–[6], have had shortcomings in their ability to achieve. High system-level tests developed with automated unit tests have become both costly and complex, thereby spurring a discussion if the technique is applicable for anything but the low system-level testing, for which it was developed [7]. Furthermore, R&R techniques, which were developed for automation of system-level tests, are instead limited by being fragile to GUI layout and API change. Limitations that in the worst case have caused entire auto- mated test suites to become inept [8]. Hence, the previous techniques have shortcomings in terms of flexibility, simplicity and robustness to make them long-term viable. However, in this case study we show that VGT can over- come these limitations. Hence, showing that VGT has the capability to automate and perform industrial grade test cases that previously had to be performed manually, with equal or even greater fault finding ability, at lower cost. Capability provided by the technique’s use of image recognition that, in combination with scenario based scripts, allow VGT tools to interact with any graphical object shown on the computer monitor, i.e. allowing VGT scripts to emulate a human user. In addition, the study presents the practitioners’ views on using the technique, e.g. benefits, problems and limitations, when performed with the open source tool Sikuli [9]. Consequently, this work shows that VGT works for testing of real-world systems when performed by practitioners facing real-world challenges such as refactoring and maintenance of the SUT. The specific contributions of this work therefore include, 1) An account on how the transition to VGT was success- fully conducted by industrial practitioners for a real- world system. 2) The industrial practitioners experiences and perception on the use of VGT. 3) Qualitative and quantitative data on costs, challenges, limitations and solutions that were identified during the VGT transition project. 2013 IEEE Sixth International Conference on Software Testing, Verification and Validation 978-0-7695-4968-2/13 $26.00 © 2013 IEEE DOI 10.1109/ICST.2013.14 56