ADAPTIVE FOREGROUND OBJECT EXTRACTION FOR REAL-TIME VIDEO SURVEILLANCE WITH LIGHTING VARIATIONS Hui-Chi Zeng and Shang-Hong Lai Department of Computer Science, National Tsing Hua University, Hsinchu, Taiwan {mr944327,lai}@cs.nthu.edu.tw ABSTRACT In this paper we present an adaptive foreground object extrac- tion algorithm for real-time video surveillance. The proposed algorithm improves the previous Gaussian mixture background models (GMMs) by applying a two-stage foreground/back- ground classification procedure to remove the undesirable sub- traction results due to shadow, automatic white balance, and sudden illumination change. The traditional background sub- traction technique usually cannot work well for situations with lighting variations in the scene. In the proposed two-stage classification, an adaptive classifier is applied to the foreground pixels in a pixel-wise manner based on the normalized color and brightness gain information. Secondly, the remaining foreground candidate pixels are grouped into regions and the corresponding background regions are compared to check if they are foreground regions. Experimental results on some real surveillance video are shown to demonstrate the robust- ness of the proposed adaptive foreground extraction algorithm under a variety of different environments with lighting varia- tions. Index Terms— Real-time, surveillance, background sub- traction, foreground extraction, lighting variation 1. INTRODUCTION The main goal of video surveillance is to detect the fore- ground objects, and background subtraction is the most fun- damental and common approach to achieve this goal. In re- cent years, several different background subtraction techniques are presented. Tuzel, et al. [1] used a Bayesian approach to background modeling. They defined each pixel as a mix- ture of multivariate Gaussian distributions and estimated the means and covariances of all Gaussian functions from a pe- riod of background video frames. Elgammal et al. [2] pro- posed a non-parametric model for background subtraction. The recent samples of intensity values for each pixel are used to compute the non-parametric probability density function. The drawback of this method is that it requires a consider- able amount of memory to store the probability density func- tions. Stauffer and Grimson [3] proposed to use a mixture of Gaussian functions to model the intensity distribution of each Fig. 1. The system flow chart. background pixel, and the background model can be gradu- ally adapted to the temporal intensity changes. After background subtraction, the subtracted non-back- ground pixels include the foreground objects and background pixels with intensity changes caused by lighting variations or auto white balance. Some shadow detection methods have been proposed in the past. In Porikli and Thornton’s work [4], they apply a shadow weak classifier as a pre-filter first, then model the selected shadow pixels using multivariate Gaus- sians. Huang et al. [5] first segmented each frame into regions based on motion similarity. The intensities of the shadow re- gions are assumed to be similar to those of the corresponding background regions by a scale. They estimate the scale to de- termine if a region belongs to a shadow region. Elgammal et al. [2] used the chromaticity coordinates r,g and the ratio of the lighting descent information for shadow detection. Tian et al. [6] presented a normalized cross-correlation algorithm for shadow removal, but it is time-consuming and it can not work well with homogeneous regions. In many cases, the lighting changes or the auto white bal- ance function is the video camera makes the background mod- eling very difficult, thus leading to unsatisfactory background subtraction results. In the proposed foreground extraction al- gorithm, as shown in Figure 1, we employ the mixture of Gaussians approach [3] to model the background, followed by a proposed two-stage procedure for classifying foreground and background pixels under lighting variations. The first step of our algorithm involves using a classifier to pixel-wisely I  1201 1424407281/07/$20.00 ©2007 IEEE ICASSP 2007