IEEE TRANSACTIONS ON INTELLIGENT TRANSPORTATION SYSTEMS, VOL. 9, NO. 3, SEPTEMBER 2008 425 Learning, Modeling, and Classification of Vehicle Track Patterns from Live Video Brendan Tran Morris, Student Member, IEEE, and Mohan Manubhai Trivedi, Senior Member, IEEE Abstract—This paper presents two different types of visual activity analysis modules based on vehicle tracking. The highway monitoring module accurately classifies vehicles into eight differ- ent types and collects traffic flow statistics by leveraging track- ing information. These statistics are continuously accumulated to maintain daily highway models that are used to categorize traffic flow in real time. The path modeling block is a more general analysis tool that learns the normal motions encountered in a scene in an unsupervised fashion. The spatiotemporal motion charac- teristics of these motion paths are encoded by a hidden Markov model. With the path definitions, abnormal trajectories are de- tected and future intent is predicted. These modules add real- time situational awareness to highway monitoring for high-level activity and behavior analysis. Index Terms—Anomaly detection, comparative flow analysis, highway efficiency, real-time tracking analysis, trajectory learning and prediction, vehicle type classification. I. I NTRODUCTION A KEY GOAL of situational awareness research is to understand the interactions and behaviors present in a scene. This scene awareness is particularly important for visual surveillance systems that must continually monitor a site. Large amounts of data are generated, making it infeasible for a human to accurately process. Activity analysis systems can be employed to filter out relevant data, focusing attention where it is needed most. Highway traffic management is an important field requiring up-to-date data delivered in real time along with historical data on traffic conditions to design effective control strategies. In California, inductive loop sensors deliver counts (number of vehicles to cross a loop) and occupancy (average fraction of time a vehicle is over a loop) every 30 s from locations all over the state, providing a large data infrastructure. Unfortunately, only about 60% of California loop detectors supply usable data, and they are costly to maintain. Cameras offer an attractive substitute for loops since they can be unobtrusively deployed on the side of a highway and can also be used for other monitoring applications. In addition to providing traffic measurements equivalent to loop detectors, using video to track vehicles in Manuscript received June 14, 2007; revised October 16, 2007 and December 24, 2007. This work was supported in part by the Technical Support Working Group and in part by the University of California Discovery Grant. The Associate Editor for this paper was N. Papanikolopoulos. The authors are with the Computer Vision and Robotics Research Labo- ratory, University of California, San Diego, La Jolla, CA 92093-0434 USA (e-mail: b1morris@ucsd.edu; mtrivedi@ucsd.edu). Color versions of one or more of the figures in this paper are available online at http://ieeexplore.ieee.org. Digital Object Identifier 10.1109/TITS.2008.922970 a scene reveals added information that is difficult to obtain using loop detectors such as origin–destination (OD) maps, travel time, and vehicle type classification. This added analysis provides a more complete highway picture than can currently be provided by loop detectors, allowing the construction of better traffic control strategies. In addition to collecting traffic statistics, it would be ad- vantageous to have a method of automatically extracting the expected highway behavior. This is particularly important when setting up a large camera network, where it is prohibitive to define every behavior, or when using pan–tilt–zoom (PTZ) cameras, where the view can drastically change, or for mon- itoring a variety of traffic scenes without tedious supervision. The underlying structure of roads constrains motion and can be leveraged to automatically build up behavior models through careful observation over time. By generating the models from the data, the learned behaviors better reflect what is actually occurring in a scene rather than what is expected. Further- more, the models allow prediction and detection of unusual or abnormal events. Without a priori knowledge, activity analy- sis is possible in an arbitrary scene just through tracking of objects. This paper present two different traffic situational aware- ness systems. The first system is the visual VEhicle Classifier and Traffic flOw analyzeR (VECTOR) [1] module for robust real-time vehicle classification, traffic statistic accumulation, and highway modeling for flow analysis. The second activity analysis module introduced is the path behavior block, which builds a probabilistic scene motion model in an unsupervised manner for activity analysis. This process automatically defines the traffic lanes without manual specification and is used to detect anomalous trajectories and unusual actions, as well as generate long-term path prediction. The efficacy of these be- havior modules are demonstrated through analysis of simulated and real-world data. II. RELATED RESEARCH A. Highway Analysis Highway analysis requires robust detection of vehicles and tracking. With these two basic tasks, a number of other cal- culations can be performed, such as vehicle classification, extraction of traffic flow parameters, congestion detection, or a number of other measurements that are useful for traffic management. A major research effort is to build large-scale systems that are able to effectively cover miles of road [2]. Key hurdles associated with this system realization are adaptation to a wide variety of changing environmental conditions [3], 1524-9050/$25.00 © 2008 IEEE Authorized licensed use limited to: IEEE Xplore. Downloaded on January 21, 2009 at 14:45 from IEEE Xplore. Restrictions apply.