voice of evidence 10 IEEE SOFTWARE Published by the IEEE Computer Society 0740-7459/08/$25.00 © 2008 IEEE Editor: Forrest Shull n Fraunhofer Center for Experimental Software Engineering, Maryland n fshull@fc-md.umd.edu Improving Evidence about Software Technologies A Look at Model-Based Testing Arilo Dias Neto, Rajesh Subramanyan, Marlon Vieira, Guilherme Horta Travassos, and Forrest Shull A rich body of experiences hasn’t yet been published on all the software development techniques researchers have proposed. In fact, by some estimates, the techniques for which we do have substantial experi- ence are few and far between. When we started looking at the evidence on model-based testing (MBT), we thought we’d come across some strong studies that showed this approach’s capabili- ties compared to conventional testing techniques— this wasn’t the case. However, we can still extract some useful knowledge and also discuss some issues that are relevant to other software technologies with similar types of evidence. A new testing paradigm MBT approaches help automatically generate test cases using models extracted from software ar- tifacts. 1 MBT promises to greatly affect how we build software as well as the level of confidence developers can have that their software meets its requirements—especially those related to safety, reliability, and performance. This promise is based on the fact that MBT approaches extract relevant testing information directly from the models, which can automate tasks that fallible humans usually undertake and allow more complete and correct test suites. The technical literature often highlights this goal. 1–3 MBT has been studied since at least 1976, when Chittoor V. Ramamoorthy and his col- leagues described a seminal approach for test-data generation. 4 Given a program graph for a Fortran program, their approach identifies a set of paths that satisfy some given testing criteria. Since this development, researchers have proposed and re- fined several different MBT approaches. According to the literature, certain specialized domains use MBT. A review of available MBT case studies, for example, highlighted its use in hardware or embedded systems—namely, processor architec- tures, communication protocols, and smart-card applications, as well as operating systems and lan- guage specifications. 5 However, our (admittedly anecdotal) experi- ence working with software development teams in other domains is that most developers don’t view MBT as a mainstream approach. It’s inter- esting to ask why not, and whether the available evidence has some light to shed on MBT’s broader usefulness. A broad view We could find no prior formal survey of MBT ap- proaches in the literature. To characterize how much evidence existed on MBT, we conducted a systematic review of the published work in this area through mid-2006. (See the details of our re- view in the Web appendix; go to www.computer. org/software and click on this issue’s Web Extra.) The initial search returned 406 papers, which we reviewed manually. Once we removed clearly irrel- evant and duplicate papers, 202 papers remained. Motivated by the popularity of the Unified Mod- eling Language (UML) among development teams that we work with, we selected as our priority for initial analysis all papers describing MBT approaches that are useful for UML models, and n