Abstract In this paper, we propose a fingerprint analysis algorithm based on using product manifolds to create robust signatures for individual targets in motion imagery. The purpose of target fingerprinting is to re-identify a target after it disappears and then reappears due to occlusions or out of camera view and to track targets persistently under camera handoff situations. The proposed method is statistics-based and has the benefit of being compact and invariant to viewpoint, rotation, and scaling. Moreover, it is a general framework and does not assume a particular type of objects to be identified. For improved robustness, we also propose a method to detect outliers of a statistical manifold formed from the training data of individual targets. Our experiments show that the proposed framework is more accurate in target re- identification than single-instance signatures and patch- based methods. 1. Introduction Large amounts of video data have become more available to defense and security analysts as unmanned aerial vehicles (UAVs) with sensor payloads are used in many ISR (intelligence, surveillance, and reconnaissance) operations and large numbers of networked cameras are used for urban surveillance. Analysts are looking for significant events/targets that may be of importance for their surveillance missions. However, most of this video footage will be eventless and therefore of no interest to the analysts responsible for checking it. If a computer system could scan the videos for potential events/targets of interest, it would greatly lessen the analysts’ workload, allowing them to focus only on the events of possible importance. Recent video surveillance paradigms for multi-object detection and tracking [15, 23] in video sequences are developed as an integrated framework to not only spot unknown targets and track them, but also handle target reacquisition and target handoff to other cameras [4, 13, 19, 24]. A common process for these paradigms can be divided into four steps: detection of objects of interest, extraction of features or signatures, dimensionality reduction of the feature space, and target template or signature matching. A variety of different tracking or fingerprinting techniques have been proposed to address one or more of these key capabilities. The tracking problem can be very challenging under unconstrained environment [7], because the target may change its appearance quickly, due to lighting conditions and frame rate, there might be other objects with similar appearance, and the background could be cluttered. In addition, occlusions and other nuisance factors of targets can further add the complexity to the problem [1]. Similar to single-object tracking, many existing multiple-object tracking methods are feature-based [15, 23] or image- patch based [17]. The latter applies sparse representation to predefined templates and random projection to speed up the solutions. Using motion and spectral clustering of trajectories instead of feature extraction is an alternative approach for tracking [10]. To handle target exiting and reappearing in videos or camera handoff situations, there are several methods working towards a unified framework of detection, tracking and identifying targets. For example, [4] demonstrates tunnel surveillance applications and shows the ability to follow vehicles through tunnels with non- overlapping cameras. Other methods that handle multi- camera networks are [13] for person re-identification, [19] for vehicles, and [24] for wide-area surveillance settings. Compact fingerprints are extracted from targets in these methods, such as Haar-features [4], covariance [24], implicit shape models, and SIFT features [13]. To handle significant scale variations and partial occlusions, [16] detects and tracks an object from a training video sequence by matching salient components of local descriptors and salient locations on a new image. In this paper, we present a video exploitation framework to spot unknown targets, track them, segment the target blobs to obtain useful features, and then create signature fingerprints for the targets so that they can be reacquired. We focus on developing a manifold-based framework of robust and compact signature fingerprinting for the target re-identification task. Fingerprinting is the acquisition of a distinct description of an object of interest Manifold-based Fingerprinting for Target Identification Kang-Yu Ni, Terrell N. Mundhenk, Kyungnam Kim, Yuri Owechko HRL Laboratories 3011 Malibu Canyon Road, Malibu, CA 90265 {kni,tnmundhenk,kkim,yowechko}@hrl.com 978-1-4673-1612-5/12/$31.00 ©2012 IEEE 1