Consensus of Regression for Occlusion-Robust Facial Feature Localization Xiang Yu 1 , Zhe Lin 2 , Jonathan Brandt 2 , and Dimitris N. Metaxas 1 1 Rutgers University, Piscataway, NJ 08854, USA 2 Adobe Research, San Jose, CA 95110, USA Abstract. We address the problem of robust facial feature localization in the presence of occlusions, which remains a lingering problem in facial analysis despite intensive long-term studies. Recently, regression-based approaches to localization have produced accurate results in many cases, yet are still subject to significant error when portions of the face are occluded. To overcome this weakness, we propose an occlusion-robust regression method by forming a consensus from estimates arising from a set of occlusion-specific regressors. That is, each regressor is trained to estimate facial feature locations under the precondition that a particular pre-defined region of the face is occluded. The predictions from each re- gressor are robustly merged using a Bayesian model that models each re- gressor’s prediction correctness likelihood based on local appearance and consistency with other regressors with overlapping occlusion regions. Af- ter localization, the occlusion state for each landmark point is estimated using a Gaussian MRF semi-supervised learning method. Experiments on both non-occluded and occluded face databases demonstrate that our approach achieves consistently better results over state-of-the-art meth- ods for facial landmark localization and occlusion detection. Keywords: Facial feature localization, Consensus of Regression, Occlu- sion detection, Face alignment. 1 Introduction Facial feature localization is a longstanding active research topic due to its wide applicability in computer vision and graphics [2,4,8,20,26,33]. Accurate local- ization is crucial for many applications, including automated face editing, face recognition, tracking, and expression analysis. Recent state-of-the-art methods such as [2,26] have achieved impressive results, not only on near-frontal faces but also faces in the wild. Despite these advances, the problem remains challenging due to large viewpoint variation, severe illumination conditions, various types of occlusions, etc. Early successes in facial feature localization, epitomized by the Active Shape Model(ASM) [7] and Active Appearance Model(AAM) [6,17], are characterized by a parametric template that is fit to a given image by optimizing over the template’s parameter space. Although effective for many cases, these parametric D. Fleet et al. (Eds.): ECCV 2014, Part IV, LNCS 8692, pp. 105–118, 2014. c  Springer International Publishing Switzerland 2014