Noname manuscript No. (will be inserted by the editor) Discriminative Extended Canonical Correlation Analysis for Pattern Set Matching Ognjen Arandjelovi´ c the date of receipt and acceptance should be inserted later Abstract In this paper we address the problem of matching sets of vectors embedded in the same input space. We propose an approach which is motivated by canonical correlation analysis (CCA), a statistical technique which has proven successful in a wide variety of pat- tern recognition problems. Like CCA when applied to the matching of sets, our extended canonical correlation analysis (E-CCA) aims to extract the most similar modes of variabil- ity within two sets. Our first major contribution is the formulation of a principled framework for robust inference of such modes from data in the presence of uncertainty associated with noise and sampling randomness. E-CCA retains the efficiency and closed form computabil- ity of CCA, but unlike it, does not possess free parameters which cannot be inferred directly from data (inherent data dimensionality, and the number of canonical correlations used for set similarity computation). Our second major contribution is to show that in contrast to CCA, E-CCA is readily adapted to match sets in a discriminative learning scheme which we call discriminative extended canonical correlation analysis (DE-CCA). Theoretical con- tributions of this paper are followed by an empirical evaluation of its premises on the task of face recognition from sets of rasterized appearance images. The results demonstrate that our approach, E-CCA, already outperforms both CCA and its quasi-discriminative coun- terpart constrained CCA (C-CCA), for all values of their free parameters. An even greater improvement is achieved with the discriminative variant, DE-CCA. Keywords Set · Matching · Vectors · Principal · Angles 1 Introduction Central to any applied problem of pattern recognition is the issue of how the entities of inter- est should be represented. A numerical description based on readily measurable quantities Centre for Pattern Recognition and Data Analytics (PRaDA) Deaking University Geelong 3216, VIC, Australia E-mail: ognjen.arandjelovic@gmail.com Tel: +61 (0)3 522-73079