Fusion of Visible, Infrared and 3D LADAR Imagery ∗ David A. Fay, Allen M. Waxman, Jacques G. Verly, Michael I. Braun, Joseph P. Racamato, and Carl Frost M.I.T. Lincoln Laboratory Sensor Exploitation Group Lexington, MA 02420 fay@ll.mit.edu ∗ This work was sponsored by the Defense Advanced Research Projects Agency under Air Force contract F19628-00-C-0002. Opinions, interpretations, conclusions, and recommendations are those of the authors and are not necessarily endorsed by the United States Government. Abstract - We have extended our previous capabilities for fusion of multiple passive imaging sensors to now include 3D imagery obtained from a prototype flash ladar. Real-time fusion of SWIR + uncooled LWIR and low-light visible + LWIR + 3D LADAR is demonstrated. Fused visualization is achieved by opponent-color neural networks for passive image fusion, which is then textured upon segmented object surfaces derived from the 3D data. An interactive viewer, coded in Java3D, is used to examine the 3D fused scene in stereo. Interactive designation, learning, recognition and search for targets, based on fused passive + 3D signatures, is achieved using Fuzzy ARTMAP neural networks with a Java-coded GUI. A client-server, web-based communication architecture, enables remote users to interact with fused 3D imagery via a wireless palmtop computer. Keywords: Sensor fusion, image fusion, real-time processing, data mining, target recognition, ladar, range data, 3D models. 1 Background Sensor operators on commercial and military platforms need to quickly make decisions based on information from multiple sources of imagery. As the number of imaging sensors increases, the variety of data increases, but so does the amount of data the operators need to process. Fusing the imagery from the different sensors, while enhancing the complementary information, can decrease the operator’s workload and improve performance by increasing target pop- out. The work presented here addresses the issue of combining imagery from passive sensors (low-light visible, SWIR, and LWIR) with range data from an active imaging ladar, to support 3D fused interactive visualization. In addition, the fused results provide input to an interactive target learning, recognition, and tracking system that can help cue an operator to potential targets of interest. We build upon work that we have presented over the last several years focusing on the real-time fusion of low-light visible and thermal infrared imagery [1-13]. We have also demonstrated the modification of these techniques to the fusion of imagery from up to six different bands, ranging from the visible to SWIR, as well as fusion of visible, infrared and SAR (synthetic aperture radar) imagery [14-17]. The addition of range imagery to the fused low-light visible and infrared imagery provides 3D spatial context, greatly enhancing scene understanding. Prior to our introduction of opponent-color image fusion, other methods of image fusion were based upon maximizing image contrasts across multiple scales via pixel comparisons and image blending [18-22]. Human factors testing has shown that the resulting gray-scale fused images do not provide for the same degree of target pop-out as our color fused results [13,23]. There are other color fusion methods that have also shown improvement over the gray-scale fusion methods for target detection, but they do so at the cost of overall visual quality [13,23,24]. Our paper is organized as follows: after the sensors and computing hardware used are described, the biological motivations and image fusion system architectures are introduced. Range image cleaning and 3D model generation methods are explained, followed by examples of multi- sensor fusion for visualization and as input to the target learning and recognition system. 2 Sensors and computing hardware Our real-time multi-sensor imaging platform is shown in Figure 1. On the left is the brassboard active imaging ladar, developed at Lincoln Laboratory [25,26], which measures range from the sensor to the scene being imaged. The ladar illuminates the scene with a 30-µJ frequency-doubled (532 nm) µchip laser, and then detects the reflected laser light with a 4x4 array of Silicon Geiger-mode avalanche photodiode detectors (APDs), which are sensitive enough to