Empirical Research in Software Process Modeling: A Systematic Literature Review Xu Bai Dept. of Computer Science and Engineering Southern Methodist University Dallas, TX USA xbai@smu.edu He Zhang National ICT Australia University of New South Wales Australia He.Zhang@nicta.com.au LiGuo Huang Dept. of Computer Science and Engineering Southern Methodist University Dallas, TX USA lghuang@smu.edu Abstract—Recognized as one of the powerful technologies in software process engineering, Software Process Modeling (SPM) has received significant attention over the last three decades. Although empirical research plays a critical role in software engineering, the state-of-the-practice of empirical research in SPM has not been systematically reviewed. This paper serves as a status report of the assessment of empirical research in SPM by analyzing all refereed studies that were published in relevant venues from 1987 to 2008 using systematic review methodology. The primary findings indicate that in current SPM-related empirical studies, (1) software process management and improvement (SPI) was not yet the most popular primary research objectives; (2) exploratory empirical research methods, e.g., case study and action research, were dominantly used; (3) there were common issues in empirical research reports in terms of following rigorous reporting guidelines. Based on the review results, we also suggest the future needs for empirical research in SPM, in terms of research topics, SPM techniques, the strengths of research methodology and the rigors of empirical studies. Keywords-Software process modeling and simulation, empirical research, systematic literature review I. I NTRODUCTION Software Process Modeling (and Simulation) (SPM) refers to the activities in creating abstract representations of the methodology, design or definition of the software process, i.e., the software development process models [1]. SPM research, which includes both static modeling and dynamic (simulation) modeling research, provides the packages of techniques and powerful tool sets that leverage the plan- ning, understanding, managing and improving systems and software processes. Empirical research in SPM plays a critical role by gathering the current best evidence from SPM research and practices, which can be integrated into the software process management and improvement practices. It is particularly useful to understand the best practices and lessons learned in applying various SPM techniques, and the strength of their empirical evidence. Systematic Literature Review (SLR) is a key research method to gather and analyze all evidence available on specific topics. However, such secondary study has not been systematically reported in SPM-related empirical studies yet, especially in analyzing the strength of empirical evidence. Our research objective is to report the state-of-the- practice of empirical research in SPM in the methodological perspectives of empirical research, and provide convicting and illuminating supports for its future improvements. This paper presents an SLR of SPM-related empirical studies published in domain specific venues including 7 journals and 6 conference proceedings, plus other relevant studies found by searching across online digital libraries, in the period of 1988 to December 2008. The SLR results are based on data collected/synthesized from 43 SPM-related empirical studies, finally selected from 326 relevant studies. The reminder of this paper is organized as follows. Section II explains our review methodology. Section III presents our review results and discusses our findings based on the data we have collected in the review. Section VI concludes the paper and envisions the direction of future research. II. RESEARCH METHOD This study followed Kitchenham and Charters’s method- ological guidelines for SLR [2]. It went through 8 major steps of the review process including Define Research Questions, Develop Search Strategy, Develop and Update Review Protocol, Find Relevant Studies, Review and Extract Data, Analyze and Synthesize Data, Quality Assessment and finally Format and Compile Report (some steps were performed simultaneously). Due to the space limitation, we only reported the most important steps in the following sections. A detailed description of our research methods, including study inclusion and exclusion criteria, quality assessment checklists etc. can be found in [3]. A. Research Questions Based on the research objectives in Section I, our research was guided by the following four questions about the empirical research in SPM: RQ.1 What are the primary research objectives? RQ.2 What SPM techniques have been used or evaluated? RQ.3 What empirical research methods and data analysis techniques have been employed? RQ.4 How rigorous are the studies (with regard to the empirical research guidelines, e.g., the guidelines for empirical research in software engineering [4])?