Dualband FLIR fusion for automatic target recognition Lipchen Alex Chan * , Sandor Z. Der 1 , Nasser M. Nasrabadi 2 US Army Research Laboratory, Attn: AMSRL-SE-SE, 2800 Powder Mill Road, Adelphi, MD 20783, USA Received 22 August 2001; received in revised form 9 July 2002; accepted 21 September 2002 Abstract We investigate the potential benefits of fusing two bands of forward-looking infrared (FLIR) data for target detection and clutter rejection. We propose a similar set of neural-based clutter rejecters and target detectors, each of which consists of an eigenspace transformation and a simple multilayer perceptron. The same architecture is used to operate on either single band or dualband FLIR input images, so that the net effects of dualband fusion can be demonstrated. When the dualband inputs are used, the component bands are combined at either pixel or feature level, thus providing insight into methods of performing data fusion in this particular application. A large set of real FLIR images is used in two series of experiments, one for clutter rejection tasks and the other for target detection tasks. In both series, the results indicate that the dualband input images do improve the performance of the clutter rejecters and target detectors over their single band counterparts. On the other hand, results of the pixel and feature level fusions are quite similar, suggesting that dimensionality reduction by the eigenspace transformation can be performed independently on the two bands. Ó 2002 Elsevier Science B.V. All rights reserved. Keywords: Data fusion; Target detection; Eigen-target; Multilayer perceptron; FLIR imagery 1. Introduction In a general sense, data fusion involves certain means and tools to combine data from different sources that may include (1) multiple spectral channels of a single sensor, (2) images taken at different times by the same sensor, or (3) data collected by different sensors at the same time [1]. Usually, the data fusion can be performed at the pixel (measurements or signals) level, feature (attributes) level, or decision (rules) level, but the fusion of information may also occurs at two or more levels concurrently [2]. We employ the framework of data fu- sion to improve the performance of an automatic target recognition (ATR) system. We propose to combine the information from a forward-looking infrared (FLIR) dualband system consisting of mid-wave (MW, 3–5 lm) and long-wave (LW, 8–12 lm) channels. The data fusion is performed at the pixel level (before an eigenspace transformation is performed) and at the feature level (after the individual bands were separately trans- formed). The accuracy and computational cost of data fusion at these two different stages is measured, and compared to the performance of their component bands at comparable complexity. Although an ATR system usually includes many al- gorithmic components, such as preprocessing, detection, clutter rejection, segmentation, feature extraction, classification, prioritization, tracking, and aim-point se- lection [3], we are only interested in the target detection and clutter rejection stages in this paper. Fig. 1 shows these two stages, together with the subsequent classifi- cation module, within an ATR system. To avoid omit- ting real targets, a target detector must accept a non- zero false alarm rate. For instance, the detector in Fig. 1 has found the correct target, but has also selected a number of background regions as potential targets (false alarms or clutter). To enhance the performance of the system, an explicit clutter rejecter may be added to reject most of the false alarms or clutter produced by the de- tector, while eliminating only a few of the targets. With almost clutter-free inputs, the subsequent target classi- fier could then operate faster and more accurately. The development of practical ATR systems has been impeded by a number of problems. Some technical ob- stacles include the large number of target classes and * Corresponding author. Tel.: +1-301-394-1677; fax: +1-301-394- 5357. E-mail addresses: arl04@rcn.com (L.A. Chan), sder@arl.army.mil (S.Z. Der), nnasraba@arl.army.mil (N.M. Nasrabadi). 1 Tel.: +1-301-394-0807; fax: +1-301-394-5234. 2 Tel.: +1-301-394-0806; fax: +1-301-394-5234. 1566-2535/02/$ - see front matter Ó 2002 Elsevier Science B.V. All rights reserved. PII:S1566-2535(02)00099-4 Information Fusion 4 (2003) 35–45 www.elsevier.com/locate/inffus