Gaussian Process Based Dual Latent Function Approach to Ordinal Regression ABSTRACT The Gaussian process prior formulation introduced by us in this paper learns a mapping for ordinal regression task using dual sets of latent functions. In this formulation one set of latent functions are associated with data items and the other set of latent functions are associated with entities. An entity is a term introduced by us in this work to refer to the object responsible for assigning ordi- nal labels to data items. For example in the collaborative filtering problem an entity corresponds to a user. In our work we assume that the entities cluster, and we use latent functions to having a Gaussian process prior to model these clusters. Similarly we also assume that the data items cluster and use latent functions having a Gaussian process prior to model these clusters. We learn the pa- rameters of these Gaussian processes in a discriminative learning framework by minimizing a loss function using an alternating min- imization procedure. The purpose of introducing dual sets of latent functions is to overcome the deficiency in the predictive nature of discriminative models for ordinal regression tasks while learning from less training data unlike generative models which have a good performance even while learning from less training data. Thus we evaluate the performance of our model on two problems, collabora- tive filtering and image annotation, by comparing with well known baseline methods using a generative model approach so as to un- derstand the efficacy of our model on less training data. 1. INTRODUCTION Machine learning tasks often involve scenarios of predicting vari- ables of ordinal scale, a setting referred to as ordinal regression. This kind of tasks arises frequently in many areas such as social sciences, information retrieval related to human preferences, etc. Different from standard machine learning tasks of classification and metric regression, ordinal values are discrete, finite and more im- portantly, the real distance between ordinal values is unknown. For example, diseases are graded on scales from least severe to most severe. In order to apply regression model to ordinal values, we need to make difficult decisions such as should we forget the or- dering of the values and treat them as nominal, should we apply some sort of scale and pretend the variables are interval, etc. So ordinal regression problems require different methods for standard metric regression. Seminal work on ordinal regression was done by [9] using a cumulative model, in which they rely on a specific dis- tributional assumption on the unobservable latent variables and a stochastic ordering of input space. In [5] general ranking problems are considered in the form of preference judgements. In this paper, we introduce a new term known as entity in the ordi- nal regression problem definition. An entity is a term introduced by us in this work to refer to the object responsible for assigning ordi- nal labels to data items. The Dual Function Gaussian Process Or- dinal Regression model introduced by us in this work learns a map- ping for ordinal regression task using dual sets of latent functions. In this formulation one set of latent functions are associated with data items and the other set of latent functions are associated with entities. In our work we assume that the entities cluster, and we use latent functions having a Gaussian process prior to model these clusters. Similarly we also assume that the data items cluster and use latent functions having a Gaussian process prior to model these clusters. We learn the parameters of these Gaussian processes in a discriminative learning framework by minimizing a loss function using an alternating minimization procedure. The alternating min- imization procedure requires us to compute the inverse of covari- ance matrix. In order to overcome the high computational overhead involved with the inversion of covariance matrix of a gaussian pro- cess, we use a matrix decomposition based on the principal eigen values to represent the inverse of covariance matrix. The symmetric property of the covariance matrix allows us to use a computation- ally less expensive method to compute the principle eigen vectors of the covariance matrix. The two problems for which we apply our ordinal regression ap- proach are collaborative filtering and image annotation. The term entity in the collaborative filtering problem indicates a user and in an image annotation problem it indicates an image. The role of an entity in collaborative filtering is to assign ratings belonging to ordinal categories to data items like movies. In the image annota- tion problem an entity corresponds to an image and the two ordinal categories indicate whether a word in the annotation vocabulary is present in the visual appearance of an image or whether the word is not present. Note that in the image annotation problem it is an im- age(entity), based on its visual properties that determines which or- dinal labels are assigned to the words(data items) in the annotation vocabulary. For example if an image(entity) contains snow capped mountain, the two words(data items) snow and mountain would be assigned the ordinal label indicating their presence irrespective of the manual mechanism used to annotate the image. Thus it is the entity which holds the responsibility for assigning ordinal labels on data items in our ordinal regression problem framework. Atulya Velivelli and Thomas S. Huang University of Illinois at Urbana-Champaign