Co-operative Multi-target Tracking and Classification Pankaj Kumar 1 , Surendra Ranganath 2 , Kuntal Sengupta 3 , and Huang Weimin 1 1 Institute for Infocomm Research, 21 Heng Mui Keng Terrace, Singapore 119613, kumar@i2r.astar.edu.sg, wmhuang@i2r.a-star.edu.sg, 2 National University of Singapore, 4 Engineering Drive 3 Singapore 117576, elsesr@nus.edu.sg 3 AuthenTec, Inc. Post Office Box 2719, Melbourne, Florida 32902-2719 kuntal.sengupta@authentec.com Abstract. This paper describes a real-time system for multi-target tracking and classification in image sequences from a single stationary camera. Several targets can be tracked simultaneously in spite of splits and merges amongst the foreground objects and presence of clutter in the segmentation results. In results we show tracking of upto 17 targets simultaneously. The algorithm combines Kalman filter-based motion and shape tracking with an efficient pattern matching algorithm. The latter facilitates the use of a dynamic programming strategy to efficiently solve the data association problem in presence of multiple splits and merges. The system is fully automatic and requires no manual input of any kind for initialization of tracking. The initialization for tracking is done us- ing attributed graphs. The algorithm gives stable and noise free track initialization. The image based tracking results are used as inputs to a Bayesian network based classifier to classify the targets into different categories. After classification a simple 3D model for each class is used along with camera calibration to obtain 3D tracking results for the tar- gets. We present results on a large number of real world image sequences, and accurate 3D tracking results compared with the readings from the speedometer of the vehicle. The complete tracking system including seg- mentation of moving targets works at about 25Hz for 352×288 resolution color images on a 2.8 GHz pentium-4 desktop. 1 Introduction This paper address several problems of tracking and classifying multiple targets in real-time, which can be used for behavior analysis of the moving targets. Several new ideas have been developed to solve the problem of Multi-Target Tracking (MTT) in 3D under the following assumptions: 1. Image sequences are obtained from a single stationary camera looking down into the scene. 2. The targets are moving on a ground plane and some 3D measurements on the ground and their corresponding locations in the image are available for camera calibration. In this paper the problem of MTT is formulated as an optimal feature estimation and data association problem, which has been the usual paradigm T. Pajdla and J. Matas (Eds.): ECCV 2004, LNCS 3021, pp. 376–389, 2004. c  Springer-Verlag Berlin Heidelberg 2004