The Virtues of Peer Pressure: A Simple Method for Discovering High-Value Mistakes Shumeet Baluja, Michele Covell, Rahul Sukthankar Google Research Abstract. Much of the recent success of neural networks can be attributed to the deeper architectures that have become prevalent. However, the deeper architec- tures often yield unintelligible solutions, require enormous amounts of labeled data, and still remain brittle and easily broken. In this paper, we present a method to efficiently and intuitively discover input instances that are misclassified by well-trained neural networks. As in previous studies, we can identify instances that are so similar to previously seen examples such that the transformation is visually imperceptible. Additionally, unlike in previous studies, we can also gen- erate mistakes that are significantly different from any training sample, while, importantly, still remaining in the space of samples that the network should be able to classify correctly. This is achieved by training a basket of N “peer net- works” rather than a single network. These are similarly trained networks that serve to provide consistency pressure on each other. When an example is found for which a single network, S, disagrees with all of the other N - 1 networks, which are consistent in their prediction, that example is a potential mistake for S. We present a simple method to find such examples and demonstrate it on two vi- sual tasks. The examples discovered yield realistic images that clearly illuminate the weaknesses of the trained models, as well as provide a source of numerous, diverse, labeled-training samples. 1 Introduction The recent rapid resurgence of interest in deep neural networks has been spurred by state of the art performance on vision and speech tasks. However, despite their impressive performance, the deeper architectures require enormous amounts of labeled data and are surprisingly fragile. Additionally, because the training is done through following derivatives in high-dimensional spaces and often through ad-hoc architectures and input groupings, the results can be unintelligible with surprising error modes [9, 13, 16]. The three problems of needing vast amounts of training data, being brittle, and being unintelligible are interrelated. In this paper, we address them by finding high- value mistakes that the network makes. A high-value mistake is one where the network is expected to perform correctly and does not – e.g., the input lies in the space of realistic inputs for the task and yet is misclassified. Finding where these mistakes are likely to occur yields insight into the computation of the network. These mistakes can also be used to further augment the training set to mitigate the brittle decision boundaries that may have been created with only the original training examples. Figure 1 provides a visual example of high-value and low-value mistakes for the familiar MNIST dataset.