IEEE TRANSACTIONS ON IMAGE PROCESSING 1 Multimodal Target Detection by Sparse Coding: Application to Paint Loss Detection in Paintings Shaoguang Huang, Member, IEEE, Bruno Cornelis, Member, IEEE, Bart Devolder, Maximiliaan Martens and Aleksandra Piˇ zurica, Senior Member, IEEE Abstract—Sparse representation based methods have demon- strated their superior performance in target detection tasks com- pared to more traditional approaches such as matched subspace detectors and adaptive subspace detectors. However, the existing sparsity-based target detection methods were mostly formulated for and validated on a single imaging modality (sometimes with multiple spectral bands). In many application domains, including art investigation, multimodal data, acquired by different sensors are readily available, and yet, efficient processing techniques for such data are still scarce. In this paper, we propose a sparsity- based multimodal target detection method that processes jointly the information from multiple imaging modalities in a kernel feature space, and making use of the spatial context. We develop our target detector such to be robust to errors in labelled data, which is especially important in applications like digital painting analysis, where pixel-wise manual annotations are unreliable. We apply the proposed method to a challenging application of paint loss detection in master paintings and we demonstrate its effectiveness on a case study with multimodal acquisitions of the Ghent Altarpiece. Index Terms—Sparse representation, target detection, paint loss, kernel, multiple imaging modalities. I. I NTRODUCTION D IGITAL painting analysis has made vast progress over the recent years, powered by a wide range of new image acquisition techniques [1]. Numerous tasks, such as characterization of painting style and forgery detection [2, 3], crack detection [4–7], authorship identification [8], classifi- cation of ancient coins [9], thread count analysis (of canvas supports) [10] and portraits [11], indexing of cultural heritage collections [12], colorization of historical art pieces [13], removal of canvas texture [14], source separation [15] and inpainting [16, 17], have demonstrated the great potential of digital image processing and machine learning in art investiga- tion. Multimodal imaging is now routinely employed in order to support the technical study of art works [6, 15], their restora- tion, conservation, and even presentation. Consulting different modalities of the same object often aids in uncovering regions This work was funded by the Fonds voor Wetenschappelijk Onderzoek (FWO) project: G.OA26.17N and received also funding from the Flemish Government (AI Research program). S. Huang and A. Piˇ zurica are with the Department of Telecommunications and Information Processing, TELIN-GAIM, Ghent University, 9000 Ghent, Belgium (e-mail: shaoguang.huang@ugent.be). B. Cornelis is with the Department of Electronics and Informatics, Vrije Universiteit Brussel, Brussels 1050, Belgium. B. Devolder is with the Princeton University Art Museum, Princeton, NJ 08544, USA. M. Martens is with the Department of Art, music and theatre sciences, Ghent University, Belgium. Fig. 1. Examples of paint loss in the macrophotography after cleaning. Left: original painting. Right: enlarged paint loss. Image copyright Ghent, Kathedrale Kerkfabriek; photo courtesy of KIK-IRPA, Brussels. or patterns of interest that would otherwise remain unnoticed, offers new insights and support for specific decisions that are taken during restoration treatments [17]. We address the problem of paint loss detection in digitized paintings and we formalize it as a particular instance of more general target detection from multimodal data. Our goal is thus to discriminate paint loss pixels, i.e. the target, from the non paint loss or background pixels in an automatic fashion. Paint losses in oil paintings are typically caused by abrasion or mechanical fracture and are often retouched or overpainted during numerous restoration campaigns. Modern conservation treatments typically require the removal of old varnish as well as old retouches and overpaint, revealing paint loss, such as in the examples shown in Fig. 1. The paint loss regions can vary significantly in size, from very tiny areas to larger holes or areas of missing paint, and typically have complex and irregular shapes. Detection of such paint loss areas is of great importance to conservators in estimating the extent of the damage within the painting, which is required for documentation purposes on one hand, but is also a crucial step in the virtual inpainting of the painting’s digital counterpart. The latter can act as a simulation within a decision-making process before the actual restoration. Digitized scans of works of art are often taken in different modalities during treatment, as shown in Fig. 2. This allows painting conservators to locate various areas of interest, such as overpaint and retouchings, as well as paint losses, in a more reliable way. In general, locating these areas is a very tedious procedure, especially in larger paintings, and is often only done approximately or in relatively small areas. Despite its importance, the problem of automatic paint loss detection has received little attention in the literature so far. Besides our earlier preliminary results, reported in two conference abstracts [18, 19], we are not aware