ONLINE FAILURE DETECTION AND CORRECTION FOR BAYESIAN SPARSE FEATURE-BASED OBJECT TRACKING Tewodros A. Biresaw †‡ , M. Soto Alvarez † , Carlo S. Regazzoni † † Department of Biophysical and Electronic Engineering, University of Genova Via Opera Pia 11A, 16145 Genoa, Italy ‡ School of Electronic Engineering and Computer Science, Queen Marry University of London Mile End Road, E1 4NS London, UK {tewodros,soto,carlo}@dibe.unige.it Abstract Online evaluation of tracking algorithms is an important task in real time tracking systems to detect failures. In vi- sual object tracking based on sparse features, detecting the failure of one of the feature points (corners) and correcting it will improve the performance of the tracker as a whole. In this paper a time reversed Markov chain is applied as eval- uation technique to identify the failed trackers and Partial Least Square regression is used for learning the correlation between feature points from training data set. The detected feature point trackers are recovered from the knowledge of the learned correlation model. The results are explained on a Bayesian algorithm for rigid/nonrigid 2D visual object tracking. The experimental outcomes show a global perfor- mance improvement of the tracking algorithm even in the presence of clutter. 1. Introduction Object tracking in video sequences has a variety of appli- cations in computer vision, video surveillance, human com- puter interaction, to mention a few. Unfortunately, no mat- ter how sophisticated the tracking algorithm is, one has to deal with failures due to the diverse nature of the data ac- quired by the camera. These failures are usually the result of occlusions, shadows, clutter, and missed detections. During tracking, detection of such failures is one of the most impor- tant tasks because it allows one to take recovery measures in order to maximize the tracking performance. In this a paper rigid/nonrigid object tracking using sparse features is con- sidered. In the particular case of feature point trackers, clut- This work was supported in part by the Erasmus Mundus Joint Doctor- ate in Interactive and Cognitive Environments, which is funded by the Ed- ucation, Audiovisual & Culture Executive Agency under the FPA n 2010- 0012. ter and missed detections are the most common sources of performance degradation. To minimize these failures, two basic steps have been proposed. In the First step, online evaluation technique is used to identify failures and in the second step, a correlation based method is applied in order to recover the detected failures. There exist different schemes for online performance evaluation of trackers that could be used to detect failures. One group of evaluation technique uses the features in the output of the tracker to compare to the features represent- ing the object, for instance motion and color characteristics, for making decisions [1]. Other groups use the information generated by trajectories [5]. One can use the combination of the above two methods to achieve the best failure de- tection mechanism. More details about online evaluation techniques can be found in the work from SanMiguel et al [6]. A more general framework for online tracking perfor- mance evaluation technique is presented as time reversed Markov process, Wu et al [8],[2]. In [8] it is shown that it can be applied to Particle filter, mean shift and Kanade- Lucas-Tomasi feature point tracking algorithms. The spec- ified time reversed Markov chain evaluation criteria is used for the feature based object tracker considered in this paper as a first step to improve its performance. The feature points from the object to be tracked have some form of correlations in the video sequence. This cor- relation property can be utilized as a self recovery for the failed trackers. In 3D articulated human body tracking Par- tial Least Square (PLS) regression has been proposed to learn the correlation that exists between the left and right sides of the body [9]. The learned correlation is used to predict the position of one group of the model points from the other group whenever there is a difficulty to obtain fea- tures. In a similar fashion this concept can be applied here for improving the feature point tracker as the second step. The PLS model learned among feature points is used to esti- 1