Machine Vision and Applications (1990) 3:1-11 Machine Vision and Applications 9 1990 Springer-VerlagNew York Inc. Hierarchical Image Fusion Alexander Toet Institute for Perception TNO, Kampweg 5, Soesterberg NL-3769-DE, The Netherlands Abstract: A hierarchical image fusion scheme is pre- sented that preserves those details from the input images that are most relevant to visual perception. Results show that fused images present a more detailed representation of the scene and provide information that cannot be ob- tained by viewing the input images separately. Detection, recognition, and search tasks may therefore benefit from this fused image representation. Key Words: sensor fusion, ratio of low-pass pyramid, mathematical morphology, contrast decomposition, multiresolution image representations I Introduction A hierarchical scheme is presented for the fusion of signals from multiple imaging systems. The input of the algorithm can be an arbitrary number of simulta- neously registered images. The only restriction of the method is that the input images must have some degree of spatial overlap. First, each input image is decomposed into a set of perceptually relevant pat- tern primitives. Pattern sets for the various source images are then combined to form a single set for the composit e image. Finally, the composite image is reconstructed from its set of primitives. As a result, the output of the algorithm is a composite image that preserves those details from the input images that are most relevant to visual perception. There are many imaging modes (e.g., direct view optics, television, forward-looking infrared, infra- red search and track, microwave radar, millimeter wave radar, laser radar, synthetic aperture radar, laser rangefinder, acoustic transducer array, radio frequency interferometer, etc.) in current use. Sys- Address reprint requests to: Alexander Toet, Institute for Perception TNO, Kampweg 5, Soesterberg NL-3769-DE, The Netherlands. tems that use a number of imaging systems severely increase the work load of a human operator. More- over, a human observer cannot reliably integrate visual information by viewing multiple images sepa- rately and consecutively. The integration of infor- mation across multiple human operators is nearly impossible. Thus, a system that fuses images from multiple sensors into a single image is likely to be of great practical value. The next section presents a hierarchical image fusion scheme. This scheme requires a complete description of the structure of the input images. A complete image representation can be obtained by studying the image structure over a range of scales (section 2.1). Such a multiresolution image descrip- tion can be produced by repeated application of a size-limiting filter operator of a progressively in- creasing scale (section 2.2). A hierarchical image decomposition based on local luminance contrast can be obtained by computing the ratio between successive levels in a multiresolution description (section 2.3). Both linear (section 2.4) and morpho- logical (section 2.5) filters can be used in the con- struction of a multiresolution image decomposition. Section 2.6 introduces an image fusion scheme based on a multiresolution contrast decomposition of the input images. Section 3 illustrates this fusion scheme by simulating the binocular perception of artificial input images (section 3.1) and by merging thermal and visual images (section 3.2). Finally, section 4 presents a discussion and some conclu- sions. 2 Multiresolution Image Fusion The essential problem in merging images for visual display is "pattern conservation": merging must preserve detail in the resulting composite image while not introducting spurious pattern elements that could interfere with subsequent analysis. Sim- ple methods to combine image details often create