Appearance Matching with Partial Data * Efstathios Hadjidemetriou and Shree K. Nayar Department of Computer Science Columbia University New York, NY 10027 Abstract Appearance matching methods use raw or filtered pixel brightness values to perform recognition. To expedite recognition, subspace methods are used to achieve com- pact representations of images. In many cases it is ad- vantageous to recognize an image based on only a sub- set of its pixels, for example, when a part of an image is occluded, or to expedite recognition. Currently, such subsets are selected either randomly or using heuristics. In this paper, we derive criteria for selecting the pixel subsets through a sensitivity analysis of the subspace. Based on these criteria, we propose two practical recog- nition algorithms. These algorithms were tested on a large number of images with degraded or partial data. In addition to faster recognition, our algorithms yield high recognition accuracy. 1 Introduction Appearance matching based on linear subspace meth- ods have found many important applications in com- putational vision, including, face recognition [Turk and Pentland, 1991], real-time 3D object recognition [Na- yar et al., 1996], and planar pose measurement [Krumm, 1996]. Appearance matching methods generally use im- age brightness values directly, without relying on the ex- traction of low-level cues such as edges, local shading, and texture. The success of this approach results from the fact that brightness values capture both geometric and photometric properties of the objects of interest. There are at least two reasons that motivate us to use a subset of the pixels in the image, rather than the complete image. First, if an image includes occlusion we would like to use only the incorrupted pixels for recognition. Secondly, using a subset of the pixels can enhance effi- ciency because recognition time in appearance matching is more or less proportional to the number of pixels used. A number of attempts have been made to perform recog- * This work was supported in part by ONR/DARPA MURI program under ONR Contract No. N00014-95-1-0601. Several other agencies and companies have also supported aspects of this research. nition with occluded or partial data [Murase and Na- yar, 1995a] [Moghaddam and Pentland, 1995] [Krumm, 1996] [Brunelli and Messelodi, 1993] [Leonardis and Bischof, 1996]. Though these approaches are interest- ing, none succeeds to address the underlying problems fully. The first three techniques select windows in an im- age based on ad–hoc heuristics that are not generally ap- plicable. The last two methods, at first, randomly select a small subset of pixels and then prune the subset with it- erative algorithms. However, these iterative schemes are not guaranteed to converge to the desired recognition re- sult. In addition, recognition based on very small random subsets is not generally reliable. The more general problem of using partial data has been investigated thoroughly in the context of statistics [Gauss, 1873] [Hotelling, 1944] [Ehrenfeld, 1955] [Hu- ber, 1981] [Cook and Weisberg, 1982] [Box and Draper, 1975]. These results are insightful but are limited in their applicability as they use assumptions that do not hold true in appearance matching. For instance, the data sets are assumed to be small (few pixels) and the measure- ments are assumed to be repeatable (multiple measure- ments at each pixel). In this paper, we derive several criteria for selecting sub- sets of image pixels that maximize recognition rate. This is accomplished by analyzing the sensitivity of the sub- space to image noise. Our criteria are then used to de- velop recognition algorithms that are general in their ap- plicability. The first algorithm automatically selects a square window within an image as the pixel subset. The use of such a window reduces sensitivity of recognition to occlusion. This is due to the fact that occlusion is more likely to appear in a large image rather than a small win- dow. The second algorithm judiciously selects the subset from the entire image, i.e. the pixels are not restricted to lie within a local region. Both algorithms are tested with a large number of noisy images. They demonstrate higher recognition performance when compared to algo- rithms that select pixel subsets randomly.