Neurocomputing 56 (2004) 167–185 www.elsevier.com/locate/neucom Improving classication with latent variable models by sequential constraint optimization Machiel Westerdijk, Wim Wiegerinck ∗ Department of Medical Physics and Biophysics, SNN, University of Nijmegen, P.O. Box 9101, Nijmegen 6500 HB, The Netherlands Received 1 December 2000; received in revised form 4 July 2002; accepted 1 October 2003 Abstract In this paper we propose a method to use multiple generative models with latent variables for classication tasks. The standard approach to use generative models for classication is to train a separate model for each class. A novel data point is then classied by the model that attributes the highest probability. The algorithm we propose modies the parameters of the models to improve the classication accuracy. Our approach is made computationally tractable by assuming that each of the models is deterministic, by which we mean that a data-point is associated to only a single latent state. The resulting algorithm is a variant of the support vector machine learning algorithm and in a limiting case the method is similar to the standard perceptron learning algorithm. We apply the method to two types of latent variable models. The rst has a discrete latent state space and the second, principal component analysis, has a continuous latent state space. We compare the eectiveness of both approaches on a handwritten digit recognition problem and on a satellite image recognition problem. c 2003 Elsevier B.V. All rights reserved. Keywords: Latent variable models; Semi-supervised learning; Support vector machines; PCA; Vector quantization; Image and character recognition 1. Introduction Probabilistic graphical models, such as hidden Markov models [2], sigmoid be- lief networks [16] and hierarchical mixtures of experts [12] are excellently suited to ∗ Corresponding author. Tel.: +31-24-3615040; fax: +31-24-3541435. E-mail addresses: machiel.westerdijk@cgey.nl (M. Westerdijk), wimw@mbfys.kun.nl, wimw@snn.kun.nl (W. Wiegerinck). 0925-2312/$-see front matter c 2003 Elsevier B.V. All rights reserved. doi:10.1016/j.neucom.2003.10.001