Using Q&A Websites as a Method for Assessing Systematic Reviews Bruno Cartaxo*†, Gustavo Pinto‡, Danilo Ribeiro*, Fernando Kamei±, Ronnie E. S. Santos*, F´ abio Q. B. da Silva*, S´ ergio Soares* IFPE, Brazil† UFPE, Brazil* IFPA, Brazil‡ IFAL, Brazil± {bfsc, dmr, ress, scbs, fabio}@cin.ufpe.br, gustavo.pinto@ifpa.edu.br, fernando.kenji@ifal.edu.br Abstract—Questions and Answers (Q&A) websites maintain a long history of needs, problems, and challenges that software developers face. In contrast to Q&A websites, which are strongly tied to practitioners’ needs, there are systematic reviews (SRs), which, according to recent studies, lack a connection with software engineering practice. In this paper, we investigate this claim by assessing to what extent systematic reviews help to solve questions posted on Q&A websites. To achieve this goal, we propose and evaluate a coverage method. We applied this method to a set of more than 600 questions related to agile software development. Results suggest that 12% of the related questions were covered. When considering specific agile methods, the majority of them have coverage below 50% or were not covered at all. We also identified 27 recurrent questions. I. I NTRODUCTION Questions and Answers (Q&A) websites empowered soft- ware developers to increase the pace of learning, allowing them not only to be more productive and more effective, but also more fulfilled [1], [2]. The software engineering (SE) community has long recognized the importance of these web- sites, and produced contributions related to both social aspects (e.g., personality traits [3], reputation [4], and gender [5]) and technical aspects (e.g., documentation [6], debugging [7], or even energy consumption [8]) of software development. On the other hand, there are systematic reviews 1 (SRs), which aim at synthesizing the best existing research to prac- tice [9]. Unfortunately, some researchers argue that there is a lack of connection between a significant number of systematic reviews and software engineering practice [10], [11]. In this paper, we investigate how systematic reviews are connected with SE practice. To achieve this goal, we propose a coverage method that consists of matching the findings of systematic reviews with SE related questions posted on Q&A websites. Specifically, the question we are trying to answer is: RQ. To what extent do systematic reviews cover software engineering related questions posted on Q&A websites? By coverage we mean: at least one finding of a systematic review offers knowledge that helps to solve a SE related question. By any means, however, we are not suggesting that systematic reviews should provide definitive evidence to an- swer these questions. Nevertheless, we believe that systematic 1 By Systematic Review we mean, any kind of secondary study, such as: systematic mappings, meta-analyses, and systematic literature reviews. reviews can provide valuable insights, that practitioners can use to get acquainted with possible solutions and, therefore, seek further evidence by themselves. Our method works as follows (details at Section II): (1) we extracted key findings of a SR, (2) we identified SE questions related to the systematic review, and (3) we applied a qualitative research approach to match whether the findings of the systematic review cover SE related questions. We investigate SE related questions posted on five Stack- Exchange websites. StackExchange is an umbrella of over 160 high-quality Q&A websites. We selected Q&A websites particularly relevant to SE practice (details at Section II-A). Although in its early stages, this study provides important contributions: 1) A coverage method used to assess systematic reviews using Q&A websites. 2) A preliminary study of how systematic reviews cover SE related questions. 3) A reusable dataset related to the analysis presented in this paper (available at http://bit.ly/2dMaaRJ). II. METHOD A. StackExchange Websites Selection We analyzed the official description of each one of the 160 Q&A websites and selected those that are related to SE, according to SWEBOK [12]. The five selected Q&A websites, and their characteristics are presented at Table I. We did not use StackOverflow since it is focused on specific coding issues, which is rarely the target of systematic reviews. According to Area51 [18], websites with A/Q ratio above two are considered as good, and above one are okay but need improvement. The first three listed websites were classified as good, and the remaining ones were okay but need improve- ment. However, when considering metrics such as number of visits per day and number avid users, RE and SREC were considered excellent, with 199 avid users and 1,905 visits per day, and 394 avid users and 4,384 visits per day, respectively. B. Systematic Reviews Selection Our set of systematic reviews are based on the tertiary study of Da Silva [11], that identified 120 SRs in SE. However, we excluded 88 SRs that, according to Da Silva, do not present guidelines to practitioners. Next, we excluded 8 SRs that do not report their search strings, since we need them on further