1 A Family of Experiments to Investigate the Influence of Context on the Effect of Inspection Techniques Marcus Ciolkowski University of Kaiserslautern and Fraunhofer IESE, Kaiserslautern, Germany ciolkows@informatik.uni-kl.de Forrest Shull Fraunhofer Center Maryland, USA fshull@fc-md.umd.edu Stefan Biffl Fraunhofer IESE, Kaiserslautern, Germany and Technische Univ. Wien Stefan.Biffl@tuwien.ac.a t Abstract For a growing population of researchers in software engineering, empirical studies have become a key approach of re- search. Empirical studies may be used, for example, to evaluate technologies and help to direct further research by revealing what problems and difficulties people have in practice. Without empirical studies, we have to rely only on intuition or educated opinion. Individual empirical studies often yield interesting results for their particular context, but typically this context is not de- scribed in sufficient detail to decide whether another context is similar enough to apply the conclusions of the study also there. We argue instead that families of experiments with a common framework for collecting context data are necessary in order to abstract conclusions at a useful level of detail. This paper describes a method to plan, conduct, and analyze coordinated, or concerted, families of experiments. The goal of the method is to maximize the quality and benefit of the individual empirical studies as part of the family and to mini- mize the effort for researchers by reusing experiment know-how. This is achieved by providing, for all studies of the ex- periment family, a common framework for context measurement, study preparation, material, and analysis. We apply the method to describe the planning steps for an experiment family on the influence of context on the effec- tiveness of defect reduction techniques. We focus on a particular technology, reading techniques for inspections, to in- stantiate this work. The first step of this experiment family is a broad survey of software companies on the state of the practice of inspection process and inspection techniques. The second step is to benchmark state-of-the-art inspection techniques with the par- ticipating organization’s own documents and inspection techniques. Keywords: Empirical Software Engineering, Experiment Family, Software Inspection, Meta Analysis. 1 INTRODUCTION Any software development project must answer key ques- tions such as: • What is the best life-cycle process model to choose for this particular project? • What is an appropriate balance of effort between quality assurance approaches in a project such as inspections and testing in a specific context? • What are the benefits, if any, to buy a readily available software component instead of developing it? The answers to such questions are difficult, since: (1) An- swers will vary based on characteristics of the develop- ment environment, the goals of the project, and the type of system being developed, and (2) the results of applying such technologies are human-dependent. The human-based effects in software development result in a non-deterministic behavior of development tech- niques. That is, the effects of human-based techniques cannot easily be expressed mathematically, and hence retain a degree of uncertainty. Empirical models — and with that, empirical studies — are a way to capture and describe such non-deterministic behavior. However, empirical studies are usually quite expensive, and their results cannot be extrapolated very far from their environment, or context. Moreover, it is unclear to what degree experimental results depend on their context, that is, which environmental factors influence the results. Insight into experimental results in different contexts can, for example, be gained by repeating, or replicating, ex- periments in these contexts. Experimentation and replication thus help broaden the