GUI Savvy End-to-End Testing with Smart Monkeys Birgit Hofer and Bernhard Peischl and Franz Wotawa * Technische Universit¨ at Graz Institute for Software Technology Inffeldgasse 16b/2, A-8010 Graz, Austria {hofer,peischl,wotawa}@ist.tugraz.at Abstract In this article we report on the development of a graph- ical user interface-savvy test monkey and its successful ap- plication to the Windows calculator. Our novel test monkey allows for a pragmatic approach in providing an abstract model of the GUI relevant behavior of the application under test and relies on a readily available GUI automation tool. Besides of outlining the employed test oracles we explain our novel decision-based state machine model, the associ- ated language and the random test algorithm. Moreover we outline the pragmatic model creation concept and report on its concrete application in an end-to-end test setting with a Windows Vista front-end. Notably in this specific scenario, our novel monkey was able to identify a misbehavior in a well-established application and provided valuable insight for reproducing the detected fault. 1. Introduction Most test automation approaches focus on systematic testing - repeating the same sequence of actions on the soft- ware under test to reveal unexpected behaviour. Despite many advantages, this traditional approach has a number of limitations and still misses serious customer-relevant de- fects in the software [11]. Moreover, today’s test automa- tion often access the application under test (AUT) via an ap- plication programming interface. As a result of that the code implementing the graphical user interface is rarely tested. However, to ensure that the application under test meets the customer’s quality attributes an end-to-end test is highly de- sirable. Applying test monkeys is a powerful supplement that can assist filling systematic (regression) test gaps and detecting * Authors are listed in alphabetical order. The research herein is partially conducted within the competence network Softnet Austria (www.soft-net.at) and funded by the Austrian Federal Ministry of Eco- nomics (bm:wa), the province of Styria, the Steirische Wirtschaftsf¨ order- ungsgesellschaft mbH. (SFG), and the city of Vienna in terms of the center for innovation and technology (ZIT). the defects. Monkey testing refers to the process of ran- domly exercising a software program by means of an au- tomated test tool. Unlike regression testing, test monkeys explore the software in a new way each time the test is run, consequently finding new defects. Notably test monkeys can be used early in the development process with low de- velopment and maintenance effort [7]. The authors of [11] report on successful applications and discuss the concrete application of dumb and smart monkeys as an orthogonal (and thus supplementing) techique to systematic testing. The availability of tools for automated treatment of graphical user interfaces allows for recording user interface actions and to replay these actions afterwards. Together with the ability to capture user interface elements and events dynamically these tools provide a solid technical grounding for setting up end-to-end regression tests that help to save costs and allow for improved testing. More recently, specifications of user interfaces have been used in order to generate test sequences and test data au- tomatically. These test cases can be automatically applied using the replay (to provide the test input) and dynamic cap- ture functionality (to implement the test oracle) of today’s GUI automation tools. However, almost all testing tools as well as the major- ity of systematic testing approaches rely on an implicit or explicit model. For example, the IDATG tool [3] allows for integrating GUI automation and model-based testing. How- ever, even under presence of these sophisticated tools faults that occur because of unexpected interactions might go un- detected. Consider for example the Windows calculator applica- tion. Figure 1 shows the graphical user interface of the cal- culator in scientific mode of the German version of Win- dows Vista. The calculator application is not a very compli- cated application and should be right. Unfortunately, this is not the case. For example entering a number, e.g., 57 and divide it by 0, leads to a correct response that a division by zero is not allowed. However, after calling the help applica- tion, the display of the result turns to 100, which should not happen. Although it is impossible to use this number after- wards directly for computation, such a behavior appears to AST’09, May 18-19, 2009, Vancouver, Canada 978-1-4244-3711-5/09/$25.00  2009 IEEE ICSE’09 Workshop 130