Debugging Integrated Systems: An Ethnographic Study of Debugging Practice Thomas Østerlie, Alf Inge Wang Norwegian University of Science and Technology {thomas.osterlie, alf.inge.wang}@idi.ntnu.no Abstract This paper explores how software developers debug integrated systems, where they have little or no access to the source code of the third-party software the system is composed of. We analyze the practice of debugging integrated systems, identifying five characteristics that set it apart from existing research on debugging: it spans a variety of operating environments, it is collective, social, heterogeneous, and ongoing. We draw implications of this for software maintenance research and debugging practice. The results presented in this paper are based on observations from an ethnographic study of the Gentoo OSS community, a geographically distributed community of over 320 developers developing and maintaining a software system for distributing and integrating third-party software packages with different Unix versions. 1. Introduction Software maintenance constitutes a significant factor (between 50 and 80 percent) in the total life- cycle costs of software systems [1]. Research suggests that software developers spend much of the maintenance effort simply trying to understand the software [2]. Current research is based on the premise that source code is the primary data source for understanding the software during debugging. Models of software errors proposed in the software engineering literature are based on the premise that software failures can be traced back to faults in the source code [3]. However, with increased attention on systems integration these are problematic premises. In component-based development [4], Web services and service-oriented architecture, along with information and enterprise systems integration [5], systems integrators have limited, if any, access to the source code of the integrated software. Even when integrating with open source software (OSS) components, research suggests that few systems integrators actually access the source code [6]. As such, systems integrators face the situation of having to debug systems without the source code to build an understanding of the problem upon. We therefore ask: without the source code, how do systems integrators make sense of problems when debugging integrated systems? Debugging integrated systems is largely unexplored in the research literature. The debugging process must be understood before it can be improved upon. This motivates a shift of focus from improving the debugging process, towards exploring how software developers debug integrated systems in practice. To this end, we have explored the practice of debugging integrated systems through an ethnographic study of the Gentoo OSS community. Gentoo is a geographically distributed community of volunteer systems integrators maintaining and operating a software distribution system for distributing and integrating third-party OSS with various Unix operating systems. Similar to existing studies of community-based OSS development [7], debugging is a core activity in the Gentoo community's software development process, too. The community is therefore well suited for studying the practice of debugging integrated systems. The shift of focus towards debugging practice requires that we draw upon research on practice. In this study we therefore draw upon research on practice and problems in organization science. This research shows that in real-world practice problems do not present themselves to practitioners as given [8]. Rather, problems have to be constructed from the materials of problematic situations that are puzzling, troubling, and uncertain. The process of constructing well-defined problems out of problem situations is often called sensemaking [9]. We will use sensemaking as a theoretical lens for exploring the practice of debugging integrated systems. This paper contributes to debugging research by identifying five characteristics that sets the practice of debugging integrated systems apart from existing research on debugging: it spans a variety of operating PREPRESS PROOF FILE CAUSAL PRODUCTIONS 1