1 Smart Surveillance: Applications, Technologies and Implications Arun Hampapur, Lisa Brown, Jonathan Connell, Sharat Pankanti, Andrew Senior and Yingli Tian. IBM T.J. Watson Research Center, 19 Skyline Drive, Hawthorne, NY 10532 arunh@us.ibm.com Abstract Smart surveillance, is the use of automatic video analysis technologies in video surveillance applications. This paper attempts to answer a number of questions about smart surveillance: What are the applications of smart surveillance? What are the system architectures for smart surveillance? What are the key technologies? What are the some of the key technical challenges? and What are the implications of smart surveillance, both to security and privacy? 1. Introduction Recent world events have created a shift in the security paradigm from "investigation of incidents " to "prevention of potentially catastrophic incidents ". Existing digital video surveillance systems provide the infrastructure only to capture, store and distribute video, while leaving the task of threat detection exclusively to human operators. Human monitoring of surveillance video is a very labor-intensive task. It is generally agreed that watching video feeds requires a higher level of visual attention than most every day tasks. Specifically vigilance, the ability to hold attention and to react to rarely occurring events, is extremely demanding and prone to error due to lapses in attention [12]. One of the conclusions of a recent study by the US National Institute of Justice [6], into the effectiveness of human monitoring of surveillance video, is as follows “These studies demonstrated that such a task[..manually detecting events in surveillance video], even when assigned to a person who is dedicated and well-intentioned, will not support an effective security system . After only 20 minutes of watching and evaluating monitor screens, the attention of most individuals has degenerated to well below acceptable levels. Monitoring video screens is both boring and mesmerizing . There are no intellectually engaging stimuli, such as when watching a television program.” Clearly today’s video surveillance systems while providing the basic functionality fall short of providing the level of information need to change the security paradigm from “investigation to preemption”. Automatic visual analysis technologies can move today's video surveillance systems from the investigative to preventive paradigm. Smart Surveillance Systems provide a number of advantages over traditional video surveillance systems, including • "the ability to preempt incidents -- through real time alarms for suspicious behaviors“ • "enhanced forensic capabilities -- through content based video retrieval“ • “situational awareness – through joint awareness of location, identity and activity of objects in the monitored space”. Section 2 provides a short introduction to various applications of smart surveillance systems. Section 3 discusses the architectures for smart surveillance systems. Section 4 presents the key technologies for smart surveillance systems. Section 5 briefly discusses the challenges in smart surveillance. Section 6 discusses the implications of smart surveillance technologies. Conclusions are presented in section 7. 2. Applications of Smart Surveillance In this section we describe a few applications of smart surveillance technology. In this section, we describe a few applications. We group the applications into three broad categories, real time alerts, automatic forensic video retrieval, and situation awareness. 2.1 Real Time Alerts: There are two types of alerts that can be generated by a smart surveillance system, user defined alerts and automatic unusual activity alerts. 2.1.1 User Defined Alerts: Here the system is required to recognize a variety of user defined events that occur in the monitored space and notify the user in real time, thus providing the user with an opportunity to evaluate the situation and take preventive action if necessary. Following are some typical events. 1. Generic Alerts: These are alerts which depend solely on the movement properties of objects within the monitored space. Following are a few common examples. 1. Motion Detection: This alert detects movement of any object within a specified zone. 2. Motion Characteristic Detection: These alerts detect a variety of motion properties of objects, including specific direction of object movement (entry through exit lane), object velocity bounds checking (object moving too fast). 3. Abandoned Object Alert: This detects objects which are abandoned, e.g., a piece of unattended baggage in an airport, or a car parked in a loading zone. 4. Object Removal: This detects movements of a user-specified object that is not expected to move, for example, a painting in a museum. 2. Class Specific Alerts: These are alerts which use the type of object in addition to the object’s movement properties. Following are a few common examples. 1. Type Specific Movement Detection: Consider a camera that is monitoring runways at an airport. In such a scene, the system could provide an alert