Using the Random Forest Classifier to Assess and Predict Student Learning of Software Engineering Teamwork Dragutin Petkovic 1 , Marc Sosnick-Pérez 1 , Kazunori Okada 1 , Rainer Todtenhoefer 3 , Shihong Huang 2 , Nidhi Miglani 1 , Arthur Vigil 1 1 Department of Computer Science San Francisco State University San Francisco, U.S.A. {petkovic, msosnick, kazokada, nidhimig, ahvigil}@sfsu.edu 2 Department of Computer Science and Engineering Florida Atlantic University Boca Raton, U.S.A. shihong@fau.edu 3 Department of Applied Computer Science University of Applied Science, Fulda Fulda, Germany rainer.todtenhoefer@informatik.hs-fulda.de Abstract—The overall goal of our Software Engineering Teamwork Assessment and Prediction (SETAP) project is to develop effective machine-learning-based methods for assessment and early prediction of student learning effectiveness in software engineering teamwork. Specifically, we use the Random Forest (RF) machine learning (ML) method to predict the effectiveness of SE teamwork learning based on data collected during student team project development. These data include over 100 objective and quantitative Team Activity Measures (TAM) obtained from monitoring and measuring activities of student teams during the creation of their final class project in our joint SE classes which ran concurrently at San Francisco State University (SFSU), Fulda University (Fulda) and Florida Atlantic University (FAU) . Although we have previously published RF analysis on a very limited data set, in this paper we provide the first RF analysis results done at SFSU on our full data set covering four years of our joint SE classes. These data include 74 student teams with over 350 students, totalling over 30000 discrete data points. These data are grouped into 11 time intervals, each measuring important phase of project development during the class (e.g. early requirement gathering and design, development, testing and delivery). In this paper we briefly elaborate on the methods of data collection and description of the data itself. We then show prediction results of the RF analysis applied to this full data set. Results show that we are able to detect student teams who are bound to fail or need attention in early class time with good (about 70%) accuracy. Moreover, the variable importance analysis shows that the features (TAM measures) with high predictive power (highly ranked by RF) make intuitive sense and even pointed us to measurements we did not originally expect to have high predicve importance, such as statistics on comments to source code postings. These measures can be used to guide educators and softare engineering managers to ensure early intervention for teams bound to fail. This research is funded in part by NSF TUES Grant # 1140172. Keywords—Assessment; software engineering teamwork; machine learning; education I. INTRODUCTION There is now a consensus across industry and academia that to be successful in today’s workplace, computer science students and software engineers must learn and practice effective software engineering teamwork skills. This need is evidenced by the unacceptably high incidence of failures of software projects in industry: about 9% are abandoned, about one third fail, and over half experience cost and schedule overruns. These project failures apparently stem from failures in communication, organization and teamwork aspects of software engineering [1-6, 25]. The emergence of global software development projects utilizing geographically distributed teams adds significant difficulty to overcoming these failure points. For the education community, though it is clear where the problem lies, little is known about the factors that influence actual student learning of software engineering teamwork skills or about how to objectively and quantitatively assess, monitor and predict student progress in the acquisition of these skills. This knowledge, especially the knowledge of the factors that most influence or best predict learning effectiveness, will enable educators to better and more efficiently assess and improve software engineering education and classroom practice and apply early classroom intervention when necessary. For industry, this knowledge will benefit project managers to improve software engineering project management. The Software Engineering Teamwork Assessment and Prediction (SETAP) project, led by San Francisco State University (SFSU) with collaborators at Fulda University, Germany (Fulda) and Florida Atlantic University (FAU), addressesing this need by using the Random Forest (RF) [18, 20] machine learning (ML) classification method for assessment, prediction, and most importantly discovery of factors determining the prediction of learning effectiveness of software engineering teamwork in the educational setting [13- 17, 30]. In this research the effectiveness of learning software engineering teamwork is defined as an ability of a student team: (i) to learn and effectively apply software engineering processes in a teamwork setting, and (process component) (ii) to work well in developing satisfactory software product (product component). This research was funded in part by NSF TUES Grant #1140172