Recognition of occluded objects by reducing feature interactions ☆ Kah Bin Lim, Jia Yun Wu ⁎ Mechatronics and Control Lab1, National University of Singapore, Singapore abstract article info Article history: Received 14 September 2011 Received in revised form 16 June 2012 Accepted 22 July 2012 Keywords: Occlusion recognition Appearance Geometry Spectral matching The main difficulty for the recognition of occluded objects lies in the fact that the original feature set is corrupted and no longer reliable to represent the object of interest. This corruption is caused by the interac- tions between features from different objects, denoted as feature interactions, which is a key issue addressed in our algorithm. In this paper, a local to global strategy is represented for the occlusion recognition problem, which combines the pairwise grouping and graph matching algorithms. Local appearance similarity serves as priors to reduce feature interactions, by which the performance of graph matching algorithms is improved in order to deal with the contaminated data set. With our formulation, a global decision on object recognition can be made based on locally gathered information. Experimental results show that the proposed framework can dramatically reduce incorrect matches and objects under severe occlusions can still be recognized. © 2012 Elsevier B.V. All rights reserved. 1. Introduction Occlusion is a challenging problem in computer vision and it compro- mise performances of algorithms in most of vision applications. In natural scenes, it is commonly found that objects are occluded or overlapped by other objects or surfaces, resulting in partially visible objects of interest. Although humans recognize these occluded objects effortlessly and in- stantaneously, implementation of this task on machines is far behind. In feature-based recognition systems, correspondences are found between images in a local or global manner, and then recognition is performed. However, in the case of occlusion, these methods often result in false match of features, because occlusion affects local feature detection and description directly. When detecting features in a scene, feature detectors equally re- spond to the appearance edges and occlusion boundaries. Therefore, oc- clusion cannot be detected in single image, leaving a corrupted feature set to represent the original object. An example is shown in Fig. 1, where feature points detected are marked in red dots. The original feature set of pliers is corrupted by features generated at occlusion boundaries, as circled in Fig. 1(b). Feature points detected from occluded object are no longer reliable to represent this object. In the rest of this paper, the feature set corruption by occlusion is referred to as feature interactions, because it is caused by interactions between features from different objects. As for the description of the detected feature points, local feature description methods implicitly assume that all the nearby or connected pixels around a detected feature point are from the same object. But this assumption is violated at occlusion boundaries, where information from two different physical surfaces is collected within individual neighbor- hood. In this context, detected features and their descriptions are no longer reliable to represent the original objects of interest. Thus, the corruptions of general feature detection and description would fail the later recognition task because of feature interactions. Consequently, “similar” features in two images are not necessarily leading to confident correspondences. It is safely concluded that feature interactions make the recognition of occluded objects more challenging than general recog- nition task. In this paper, a novel algorithm is proposed for occlusion recogni- tion by reducing feature interactions in terms of geometry, texture and color. Since local features alone may not be reliable to represent occluded objects, other information such as appearance relationships between features, could provide evidence to reduce feature interac- tions. In addition, a global inference of the object structure is introduced by retaining pairwise geometric consistent assignments. Our strategy focuses on making a global recognition decision based on locally gath- ered information. The rest of this paper is organized as follows. Section 2 introduces the related works in recognition and motivation for our work. Section 3 calculates the pairwise feature appearance similarity as priors to reduce feature interactions. Section 4 formulates our algorithm to recognize occluded objects. In Section 5, our algorithm is implemented under vary- ing occlusion rates and the experiments show the improvements of spectral matching algorithms to recognize occluded objects with our for- mulation. Section 6 contains the discussions and conclusions. 2. Related works Object recognition plays an important role in computer vision, and great progresses have been made in recent years. However, there are still many obstacles which have to be overcome. One typical problem Image and Vision Computing 30 (2012) 906–914 ☆ This paper has been recommended for acceptance by Matthew Turk. ⁎ Corresponding author. Tel.: +65 83147407. E-mail address: g0700814@nus.edu.sg (J.Y. Wu). 0262-8856/$ – see front matter © 2012 Elsevier B.V. All rights reserved. http://dx.doi.org/10.1016/j.imavis.2012.07.006 Contents lists available at SciVerse ScienceDirect Image and Vision Computing journal homepage: www.elsevier.com/locate/imavis