Detecting Questionable Observers Using Face Track Clustering Jeremiah R. Barr, Kevin W. Bowyer and Patrick J. Flynn Department of Computer Science and Engineering University of Notre Dame Email: {jbarr1,kwb,flynn}@nd.edu Abstract We introduce the questionable observer detection prob- lem: Given a collection of videos of crowds, determine which individuals appear unusually often across the set of videos. The algorithm proposed here detects these individ- uals by clustering sequences of face images. To provide ro- bustness to sensor noise, facial expression and resolution variations, blur, and intermittent occlusions, we merge sim- ilar face image sequences from the same video and discard outlying face patterns prior to clustering. We present ex- periments on a challenging video dataset. The results show that the proposed method can surpass the performance of a clustering algorithm based on the VeriLook face recognition software by Neurotechnology both in terms of the detection rate and the false detection frequency. 1. Introduction A potentially useful application of automatic face recog- nition technology is that of analyzing videos of crowds ob- serving the aftermath of criminal activities such as arson. Informants, accomplices or culprits may observe a collec- tion of related crime scenes; this tendency could indicate their involvement in a series of offenses. We thus call these individuals questionable observers. In contrast, we call in- dividuals that observe relatively few scenes of this nature casual observers. The distinguishing feature that separates the questionable observers from the casual observers is the percentage of videos in which they appear. We propose an algorithm for the questionable observer detection problem, which is to differentiate questionable observers from casual observers. Whereas much of the work on face recognition from video assumes that an identification algorithm has prior knowledge about the persons to recognize and typically fo- cuses on the recognition of one subject at a time, the prob- lem posed here requires the use of unsupervised classifica- tion techniques to aggregate images of the same face ob- served in different crowd videos. Identifying information Figure 1. A solution to the questionable observer detection prob- lem is a collection of face track clusters, each of which represents a distinct individual and contains face tracks from multiple videos. is lost when crowd members occlude one another, which makes both tracking and classification more difficult. Vari- ations in pose, illumination, and facial expression through- out a single video and between different videos can affect face appearance and, hence, complicate questionable ob- server detection as well. Finally, the video evidence may be recorded by camera phones or surveillance cameras and so the quality of the face image sequences can be very low. We propose an unsupervised classification algorithm for detecting questionable observers that begins by merging face image sequences, i.e. face tracks, that correspond to the same individual and come from a particular video. We then remove outlying face images from the merged face tracks based on the observations that certain head poses and facial expressions are more likely than others [2, 3]. This opera- tion reduces the influence of unrepresentative data and in- creases the homogeneity of the sampling encompassed by an individual face track. The detection algorithm subse- quently clusters the face tracks, ideally placing all of the face tracks that represent a particular individual in the same