Is it Over Yet? Learning to Recognize Good News in Financial Media Anthony Brew, Derek Greene, and P´adraig Cunningham School of Computer Science and Informatics, University College Dublin {anthony.brew,derek.greene,padraig.cunningham}@ucd.ie University College Dublin Technical Report UCD-CSI-2010-1 1 January 2010 Abstract. Until recently, tracking sentiment in news media required professional annotators to identify the polarity of individual articles so that general trends could be identified. In the work described here we use crowdsourcing to gather non-expert annotations, in conjunction with a supervised learning strategy that generalizes from the manual annota- tions to label a larger body of news articles. Our analysis of this strategy shows that, while it is effective, there are three key issues that have to be addressed: consensus, coverage, and bias. By obtaining multiple annotations for an article we can establish a consensus for the article. Alternatively we can seek only a single annotation for each article in or- der to maximize coverage, but without the benefit of a group consensus. With bias, we are not so much concerned by bias among the annotators as by the bias in the learning system which can favor the majority class. In this paper we address these three issues in the context of an analysis of media sentiment towards the Irish economic situation. 1 Introduction We are concerned with the challenge of tracking general sentiment trends on specific topics in online content. In the demonstration system discussed here we focus on sentiment in online news sources concerning the Irish economic situa- tion 2 . The insights offered from such an analysis are best explained with reference to the time-plot shown in Figure 1. This plot shows aggregate sentiment from three news sources, together with a micro-average reflecting overall sentiment. For instance, we can see that RTE, the national broadcaster (indicated by top line) is more optimistic than the other sources – at least in the news feeds ana- lyzed here. This is somewhat surprising as the Irish government frequently refer 1 This research was supported by Science Foundation Ireland (SFI) Grant Nos. 05/IN.1/I24 and 08/SRC/I1407. 2 See: http://sentiment.ucd.ie