A Hierarchical Approach to Robust Background Subtraction using Color and Gradient Information Omar Javed, Khurram Shafique and Mubarak Shah Computer Vision Lab, School of Electrical Engineering and Computer Science, University of Central Florida E-mail: {ojaved,khurram,shah}@cs.ucf.edu Abstract We present a background subtraction method that uses multiple cues to robustly detect objects in adverse conditions. The algorithm consists of three distinct levels i.e pixel level, region level and frame level. At the pixel level, statistical models of gradients and color are separately used to classify each pixel as belonging to background or foreground. In re- gion level, foreground pixels obtained from the color based subtraction are grouped into regions and gradient based sub- traction is then used to make inferences about the validity of these regions. Pixel based models are updated based on de- cisions made at the region level. Finally frame level analy- sis is performed to detect global illumination changes. Our method provides the solution to some of the common prob- lems that are not addressed by most background subtraction algorithms such as quick illumination changes, repositioning of static background objects, and initialization of background model with moving objects present in the scene. 1. Introduction All Automated surveillance systems require some mech- anism to detect interesting objects in the field of view of the sensor. Such a mechanism serves as a form of focus of at- tention. Once objects are detected, the further processing for tracking and activity is limited in the corresponding regions of the image. In vision based systems, such detection is usu- ally carried out by using background subtraction methods. These methods build a model of the scene background, and for each pixel in the image, detect deviations of pixel fea- ture values from the model to classify the pixel as belonging either to background or to foreground. This pixel based in- formation is then grouped to make a similar classification of regions in the image. Though, pixel intensity or color are the most commonly used features for scene modelling, recently some effort has been made to combine this information with edges [7]. The Background differencing methods have to deal with several problems in realistic environments. These problems have been discussed in detail by Toyama et.al [15]. Here we briefly describe some of the important problems which have not been addressed by most background subtraction al- gorithms. • Quick illumination changes : Quick illumination changes completely alter the color characteristics of the background, and thus increase the deviation of background pixels from the background model in color or intensity based subtraction. This results in a drastic increase in the number of falsely detected foreground regions and in the worst case, the whole image appears as foreground. This shortcoming makes surveillance under partially cloudy days almost impossible. • Relocation of the background Object : Relocation of a background object induces change in two different re- gions in the image, its newly acquired position and its previous position. While only the former should be identified as foreground region, any background sub- traction system based on color variation detects both as foreground. • Initialization with moving objects : If moving objects are present during initialization then part of the back- ground is occluded by moving objects. Thus many al- gorithms require a scene with no moving objects during initialization. This puts serious limitations on systems to be used in high traffic areas. • Shadows . Objects cast shadows that might also be clas- sified as foreground due to the illumination change in the shadow region. In this paper we propose solutions to the first three prob- lems. We have already presented a method to remove cast shadows [9] from a scene. Please see [13] for a detailed re- view of shadow removing algorithms. Color based background systems are susceptible to sud- den changes in illumination. Gradients of image are rela- tively less sensitive to changes in illumination and can be combined with color information effectively and efficiently to perform quasi illumination invariant background subtrac- tion. We also note that only pixel level processing is not 1