Information Sciences 375 (2017) 138–154 Contents lists available at ScienceDirect Information Sciences journal homepage: www.elsevier.com/locate/ins Judgment analysis of crowdsourced opinions using biclustering Sujoy Chatterjee a , Malay Bhattacharyya b,∗ a Department of Computer Science and Engineering, University of Kalyani, Nadia – 741235, India b Department of Information Technology, Indian Institute of Engineering Science and Technology, Shibpur, Howrah – 711103, India a r t i c l e i n f o Article history: Received 13 October 2015 Revised 1 July 2016 Accepted 14 September 2016 Available online 28 September 2016 Keywords: Judgment analysis Opinion ensemble Majority voting Biclustering a b s t r a c t Annotation by the crowd workers serving online is gaining focus in recent years in diverse fields due to its distributed power of problem solving. Distributing the labeling task among a large set of workers (may be experts or non-experts) and obtaining the final consensus is a popular way of performing large-scale annotation in a limited time. Collection of multi- ple annotations can be effective for annotation of large-scale datasets for applications like natural language processing, image processing, etc. However, as the crowd workers are not necessarily experts, their opinions might not be accurate enough. This causes problem in deriving the final aggregated judgment. Again, majority voting (MV) is not suitable for such problems because the number of annotators is limited and they have multiple options to choose. This might cause too much conflicts among the opinions provided. Additionally, there might exist annotators who randomly try to annotate (provide spam opinions for) too many questions to maximize their payment. This can incorporate noise while deriving the final judgment. In this paper, we address the problem of crowd judgment analysis in an unsupervised way and a biclustering-based approach is proposed to obtain the judg- ments appropriately. The effectiveness of this approach is demonstrated on four publicly available small-scale Amazon Mechanical Turk datasets, along with a large-scale Crowd- Flower dataset. We also compare the algorithm with MV and some other existing algo- rithms. In most of the cases the proposed approach is competitively better than others. But most importantly, it does not use the entire dataset for deriving the judgment. © 2016 Elsevier Inc. All rights reserved. 1. Introduction Crowdsourcing is one of the emerging fields that has been shown to have wide applications in the areas of data mining, machine learning, bioinformatics, etc. [17,18,25]. It provides the new opportunities to tackle diverse real-life problems using the united power of independent crowd workers. The recent popularity of online crowdsourcing services has also become useful for managing large-scale labeling tasks. The crowdsourcing model was formally introduced by Howe in 2006 [4,9]. However, Amazon Mechanical Turk (AMT) was the first crowd-powered system that appeared in 2005 to successfully solve diverse problems using the online crowd workers. Other than AMT, various crowdsourcing platforms like WikiProjects, Kick- start, Crowdfyerged, etc. are the most popular tools. These crowdsourcing platforms can be categorized into two main types ∗ Corresponding author. E-mail addresses: sujoy@klyuniv.ac.in (S. Chatterjee), malaybhattacharyya@it.iiests.ac.in, malaybhattacharyya@gmail.com (M. Bhattacharyya). http://dx.doi.org/10.1016/j.ins.2016.09.036 0020-0255/© 2016 Elsevier Inc. All rights reserved.