Emotion Annotation: Rethinking Emotion Categorization Emily ¨ Ohman 1[0000-0003-1363-7361] University of Helsinki, Finland Tampere University, Finland emily.ohman@helsinki.fi Abstract. One of the biggest hurdles for the utilization of machine learning in interdisciplinary projects is the need for annotated train- ing data which is costly to create. Emotion annotation is a notoriously difficult task, and the current annotation schemes which are based on psychological theories of human interaction are not always the most con- ducive for the creation of reliable emotion annotations, nor are they opti- mal for annotating emotions in the modality of text. This paper discusses the theory, history, and challenges of emotion annotation, and proposes improvements for emotion annotation tasks based on both theory and case studies. These improvements focus on rethinking the categorization of emotions and the overlap and disjointedness of emotion categories. Keywords: Emotion Annotation, Textual Expressions of Emotions, The- ories of Emotion. 1 Introduction Sentiment analysis has progressed along with general developments in Natural Language Processing (NLP) and machine learning in the past two decades [35], with more and more advanced models and algorithms aiding in the detection of sentiments and emotions in text. Most of these machine learning models are supervised, which means that they require large manually annotated datasets. Annotation tasks range in difficulty based on the data being annotated, the annotation scheme, and the training received by the annotators. Emotion an- notation is notoriously difficult, a notion shared by many emotion researchers particularly in the field of NLP (see e.g. [4, 7, 29, 34, 48, 50]). To the best of my knowledge, most emotion detection papers use some vari- ation of Ekman’s [19–21] six core emotions (anger, disgust, fear, happiness, sad- ness, surprise ), or Plutchik’s [43] eight core emotions (anger, anticipation, dis- gust, fear, joy, sadness, surprise, trust ). These are based on well-known psy- chological theories that have been researched extensively for decades and were therefore natural starting points for computational emotion detection. However, whether these categories are the best at describing human emotions is a question still debated in the emotion community [45] and recently whether these emotions Copyright © 2021 for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0).