Analysis and Detection of Shadows in Video Streams: A Comparative Evaluation Andrea Prati, Rita Cucchiara Dip. di Ingegneria dell’Informazione University of Modena and Reggio Emilia Modena, Italy - 41100 Ivana Miki´ c, Mohan M. Trivedi Dept. of Electrical and Computer Engineering University of California, San Diego La Jolla, CA, USA - 92037 Abstract Robustness to changes in illumination conditions as well as viewing perspectives is an important requirement for many computer vision applications. One of the key fac- tors in enhancing the robustness of dynamic scene analy- sis is that of accurate and reliable means for shadow de- tection. Shadow detection is critical for correct object de- tection in image sequences. Many algorithms have been proposed in the literature that deal with shadows. How- ever, a comparative evaluation of the existing approaches is still lacking. In this paper, the full range of problems un- derlying the shadow detection are identified and discussed. We classify the proposed solutions to this problem using a taxonomy of four main classes, called deterministic model and non-model based and statistical parametric and non- parametric. Novel quantitative (detection and discrimina- tion accuracy) and qualitative metrics (scene and object in- dependence, flexibility to shadow situations and robustness to noise) are proposed to evaluate these classes of algo- rithms on a benchmark suite of indoor and outdoor video sequences. 1. Introduction Moving object segmentation is an essential issue in many computer vision applications dealing with image se- quences. Moving shadows do, however, cause serious prob- lems while extracting moving objects, due to the misclas- sification of shadow points as foreground. Shadows can cause merging of objects, object shape distortion and even object losses (due to the shadow cast over another object). The difficulties associated with shadow detection arise since shadow and objects share two important visual features. Shadows are dark and typically differ significantly from the background and they have the same motion as the objects casting them. In literature there are many different approaches to mov- ing object segmentation from image sequences, based on inter-frame differencing, background subtraction, optical flow, statistical point classification or feature matching and tracking. However, neither motion segmentation nor change detection methods can distinguish between moving objects and moving shadows. For this reason, the efforts of com- puter vision community in finding robust shadow detection algorithms have intensified in the recent years. In this paper we present a survey of shadow detection ap- proaches, providing both a classification and a comparative evaluation of representative algorithms present in literature. This comparison will take into account both the advantages and the drawbacks of each algorithm class and will furnish a quantitative (objective) and qualitative (subjective) evalu- ation of them. In the next Section the approaches to detect shadows are organized in a taxonomy in Section 2. Each approach is de- tailed and discussed to emphasize its strengths and its limits. Section 3 presents the evaluation metrics chosen to compare the approaches and outlines their relevance, while Section 4 reports the quantitative and qualitative experimental results. Conclusions end the paper. 2. Taxonomy of shadow detection algorithms Most of the proposed approaches take into account the shadow model described in [18][22] . To account for their differences, we have organized the efforts present in lit- erature in a taxonomy. The first classification considers whether the decision process introduces and exploits uncer- tainty. Deterministic approaches use an on/off decision pro- cess, whereas statistical approaches use probabilistic func- tions to describe the class membership. Introducing un- certainty to the class membership assignment can reduce noise sensitivity by relaxing ill-posed constraints. In the statistical-based methods (as [16][5][10]) the parameter se- lection is a critical issue. The work reported in [16] is an example of the parametric approach, whereas [5][10] are examples of the non-parametric approach. Within the deterministic class (see [12][22][3][14]), an- other sub-classification can be based on whether the on/off ISBN 0-7695-1272-0/01 $10.00 (C) 2001 IEEE