Surveying Developer Knowledge and Interest in Code Smells through Online Freelance Marketplaces Aiko Yamashita Simula Research Laboratory Lysaker, Norway Email: aiko@simula.no Leon Moonen Simula Research Laboratory Lysaker, Norway Email: leon.moonen@computer.org Abstract—This paper discusses the use of freelance market- places to conduct a survey amongst professional developer’s about specific software engineering phenomena, in our case their knowl- edge and interest in code smells and their detection/removal. We present the context and motivation of our research, and the idea of using freelance marketplaces for conducting studies involving software professionals. Next, we describe the design of the survey and the specifics on the selected freelance marketplace (i.e., Freelancer.com). Finally, we discuss why freelance markets constitute a feasible and advantageous approach for conducting user evaluations that involve large numbers of software profes- sionals, and what challenges such an approach may entail. Index Terms—survey; user evaluation; developer knowledge I. I NTRODUCTION AND MOTIVATION The presence of code smells indicates that there are issues with code quality, such as understandability and changeability, which can lead to a variety of maintenance problems, including the introduction of faults [1]. In the last decade, code smells have become an established concept for patterns or aspects of software design that may cause problems for further develop- ment and maintenance of these systems [2]. Since code smells are associated to specific set of refactor- ing strategies to eliminate them, code smell analysis allows people to integrate both assessment and improvement into the software evolution process itself. Van Emden and Moonen [3] provided the first formalization of code smell detection and developed an automated code smell detection tool for Java. Other approaches for code smell detection can be found in [4– 13]. Automated code smell detection has been implemented in commercial tools such as Together 1 , Analyst4J 2 , Stan4J 3 , InCode 4 , NDepend 5 , and CppDepend 6 , and in free tools like JDeodorant 7 , and OClint 8 . Even though code smell detection and removal has been well-researched over the last decade, it remains open to debate whether or not code smells should be considered meaningful conceptualizations of code quality issues from the developer’s perspective. For example, the authors on a recent study on the lifespan of code smells in seven open source systems conclude that developers are aware but not concerned by the existence of code smells [14]. In another study, Yamashita and Moonen investigate how code smells relate to maintainability 1 http://www.borland.com/us/products/together 2 http://www.codeswat.com 3 http://wwww.stan4j.com 4 http://www.intooitus.com/products/incode 5 http://www.ndepend.com 6 http://www.cppdepend.com 7 http://www.jdeodorant.com 8 http://oclint.org characteristics considered important by professional develop- ers [15]. They found that although some concepts covered by code smells reflect maintainability properties important to developers, a considerable percentage of maintainability properties were unrelated to code smells. So, the question remains if code smells are really important to developers? If they are not, is this due to the lack of relevance of the underlying concepts (e.g., as investigated in [15]), a lack of awareness about code smells on the developer’s side, or due to the lack of appropriate tools for code smell analysis and/or removal? If tool support is lacking, what fea- tures would best support developers’ needs? Thus, to align and direct research efforts to address actual needs and problems of professional developers, we need to better understand their level of knowledge and interest in code smells. A good starting point to investigate these aspects would be an exploratory, descriptive survey. However, to conduct such a study, one needs a large enough, and representative sample of professional software engineers, which is gener- ally difficult to attain. In this paper, we suggest the use of online freelance marketplaces for obtaining such a sample. The remainder of this paper is as follows: First, we briefly discuss the challenge in Software Engineering (SE) research of accessing representative samples in controlled studies. Second, we introduce outsourcing on freelance marketplaces as an approach to overcome the “sampling” challenge. Third, we describe the design of our survey study and the specifics on the particular freelance marketplace that we selected (i.e., Freelancer.com). Finally, we discuss why the use of freelance markets constitutes an advantageous approach for conducting user evaluations involving large numbers of software profes- sionals, and what challenges such an approach may entail. II. THE CHALLENGE OF REPRESENTATIVE SAMPLES IN SE As mentioned above, our goal was to conduct a survey to investigate developer’s insights and interest in code smells. A survey is defined as: “A system for collecting information from or about people to describe, compare or explain their knowledge, attitudes and behavior” [16]. An important challenge in Software Engineering (SE) re- search is to generalize from the specific subjects, technology, tasks and systems of experiments to industrial contexts [17]. Controlled studies that are performed on small systems or in unrealistic settings suffer from threats to their external validity. 978-1-4673-6433-1/13/$31.00 c  2013 IEEE USER 2013, San Francisco, CA, USA 5