Evaluating Standard Classifiers for Detecting COVID-19 Related Misinformation Daniel Thilo Schroeder 23 , Konstantin Pogorelov 1 , Johannes Langguth 1 1 Simula Research Laboratory, Norway 2 Simula Metropolitan Center for Digital Engineering, Norway 3 Technical University of Berlin, Germany daniels@simula.no,konstantin@simula.no,langguth@simula.no ABSTRACT This paper summarises the results created through participation in the task FakeNews: Corona Virus and 5G Conspiracy of the MediaEval Multimedia Evaluation Challenge 2020. The task consists of two parts intending to detect tweets and retweet cascades that emerged during the COVID-19 pandemic and causally connect the radiation of 5G networks with the virus. We applied several well-established neural networks and machine learning techniques for the frst subtasks, namely, textual information classifcation. For the second task, the retweet cascades analysis, we rely on classifers that work on established graph features, such as the clustering coefcient or graph diameter. Our results show a MCC-score of 0.148 or 0.162 for the NLP task and 0.02 for the structure task. 1 INTRODUCTION The COVID-19 pandemic and the associated lockdown formed the basis for a multitude of false news and conspiracy myths. While a large amount of this content is limited to virtual spaces such as news portals or online social networks, some amount of par- ticularly fast-moving content has immediate consequences in the real world. The FakeNews-task[9] at the MediaEval challenge 2020 targeted the classifcation of tweets and retweet cascades of such a so-called Digital Wildfres. More precisely, the claim that COVID-19 is causally related to the radiation emitted by 5G towers, which began to spread rapidly during February 2020 and later, in early April, culminated in attacks on telecommunication staf and ar- son against 5G masts. This naturally raises the question whether Tweets promoting such dangerous misinformation can be detected automatically, either by analyzing tweet contents or the network between Twitter accounts that spread such misinformation, and the cascades created by the spread. While the analysis of tweets is an established problem, the ex- amination of retweet cascades is the subject of ongoing research. Retweet cascades can be obtained from Twitter without much efort[11], that their quality can be improved[4]. Previous work indicates that the false content spreads faster, deeper, and wider than true content[12] and that accounts that support similar ideas tend to form clusters in the Twitter graph which are sometimes referred to as echo chambers [2, 7]. While natural language processing has advanced rapidly in the last three years [3, 5], such NLP techniques rely on massive models. Similarly, retweet cascades can be analyzed using graph neural net- works, but they too tend to be computationally expensive. Our goal Copyright 2020 for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0). MediaEval’20, 14-15 December 2020, Online in this paper is to assess the performance of established methods that are both simple and computationally cheap for both subtasks. 2 SUBTASK 1: NLP-BASED DETECTION For the NLP-based detection subtask we decided to evaluate an entry-level complexity of the tweet classifcation based only on plain text content. Thus we implemented the simple out-of-box multi-class classifer based on the standard implementation of Sup- port Vector Machine (SVM) from the scikit-learn framework [8]. During our experiments, we observed that the execution of the SVM classifer for both single- and multi-class cases on the raw text data extracted from the corresponding tweeter posts resulted in worse-than-random performance. A quick look into the down- loaded training data showed that the text blocks of most tweets are heavily contaminated by non-informative character noise, emoti- cons, and various URLs. To overcome this problem, we implemented a multi-stage text cleaning procedure that performs the following steps on each text block: remove all control characters; remove all hyperlinks; glue all sequences of non-numerical and non-letter char- acters into one symbol; remove all hashtags and user references; remove all non-punctuation symbols and remove all short words. As a result, single- and multi-class SVN classifers performed more efciently, giving better than random classifcation results during the cross-validation on the provided development set. After a series of experiments, we selected the two best-performing confgurations of SVM-classifers (see Table 1 for the classifer confgurations). Table 1: NLP-based classifer parameters Run Classifer Kernel Ngram Vertorizer NLP Single-Class SVC linear (1,6) TF-IDF NLP Multi-Class SVC linear (1,12) TF-IDF 3 SUBTASK 2: GRAPH-BASED DETECTION Our graph-based detection approach is relatively straightforward and assumes that individuals spreading false messages tend to be or- ganized in so-called homophile networks. Homophile networks are more strongly connected internally than externally, which relates to echo chambers and related concepts which can lead to resis- tance to ideas coming from outside the homophile network [13]. However, because Twitter does not allow access to true retweet cascades (i.e. for each retweet list the specifc account whose ex- isting retweet caused the new retweet) but instead returns just a list of retweeters for a particular tweet, this challenge ofers the subgraphs of Twitters’ follower network that were induced by these