FUSION 2008 Tutorial Proposal: Fundamentals of the Class-Specific Method Dr. Paul M. Baggenstoss Naval Undersea Warfare Center Newport RI, 02841 401-832-8240 (TEL) p.m.baggenstoss@ieee.org January 15, 2008 Abstract The class-specific method (CSM) is an approach for con- structing classifiers with class-dependent features that blends signal processing with classifcation theory. CSM is based on the mathematical identity called the proba- bility density function (PDF) projection theorem that ex- tends classical Bayesian theory. In contrast to conven- tional classifiers, feature extraction is an integral part of the theory. Because CSM does not need a common fea- ture space, the dimensionality curse can be avoided. The tutorial covers fundamentals including generative and dis- criminative classiifers, the PDF projection theorem, class- specific modules and the chain rule. Many intuitive ex- amples are providded. Some advanced examples are also covered. 1 Overview The class-specific method (CSM) is emerging as an im- portant new generative method in signal modelling and classification. Numerous papers (see bibliography be- low), a tutorial article[1], and a conference tutorial [2] have appeared on CSM. 2 Background While the ultimate goal of all classifiers is to distinguish the various classes, the difference between generative and discriminative classifiers is the approach taken. While discriminative classifiers achieve the goal directly through construction of decision boundaries in a common feature space, generative classifiers achieve the goal indirectly by statistically modelling each class. Each method has a good argument: proponents of discriminative classi- fiers argue that it is better to estimate decision bound- aries directly while proponents of generative classifiers argue that generative classifier are embodiments of the optimal Bayesian classifier and by fully modelling each class, there is a better chance of rejecting new unseen data types. The best argument, however, is that the best classi- fiers contain elements of both generative and discrimina- tive classifiers. Recently, most attention has been paid to discrimina- tive methods. The treatment of generative methods is inadequate, not just from the point of view of attention, but from the point of view that most treatments of gen- erative classifiers are in the context of a common feature space. Within the constraints of a common feature space, discriminative classifier are often better. However, with the introduction of the PDF projection theorem (PPT) in 2000 [4], [5], [23], the situation has changed for genera- tive classifiers. The constraint of living within a common feature space has been removed and each data class can be modelled using a dedicated signal processing chain. The class-specific method (CSM) blends signal pro- cessing with classification theory using the theoretical foundation of the PDF projection theorem [5]. At times alone, but mainly when combined with discrim- inative methods, CSM can produce substantial perfor- mance improvements. The so-called curse of dimension- ality plagues conventional generative and discriminative classification methods because when restricted to a com- mon feature space, it is necessary to seek a compromise between conflicting goals. If the feature dimension is too low, critical information is lost. If the feature dimension is too high, estimation of PDFs and decision boundaries are plagued by the dimensionality issues. In CSM, because a given feature set does not need to represent all classes, the features are required only to dis- tinguish a given class from a reference hypothesis that can be chosen separately for each class. As new classes are added, no changes are required for existing models. While many methods exist for class-dependent feature extrac- tion, [6],[7], [8], none except CSM are based on a rigid general theory that is directly tied to the optimal Bayesian classifier. Thus, they are subject to approximations and restrictions, whereas CSM is a generalization of classical theory. Because CSM is a generative approach (focused 1