EUROGRAPHICS 2008 / G. Drettakis and R. Scopigno (Guest Editors) Volume 27 (2008), Number 2 The Shadow Meets the Mask: Pyramid-Based Shadow Removal Yael Shor Dani Lischinski School of Computer Science and Engineering The Hebrew University of Jerusalem Abstract In this paper we propose a novel method for detecting and removing shadows from a single image thereby obtain- ing a high-quality shadow-free image. With minimal user assistance, we first identify shadowed and lit areas on the same surface in the scene using an illumination-invariant distance measure. These areas are used to estimate the parameters of an affine shadow formation model. A novel pyramid-based restoration process is then applied to produce a shadow-free image, while avoiding loss of texture contrast and introduction of noise. Unlike previous approaches, we account for varying shadow intensity inside the shadowed region by processing it from the interior towards the boundaries. Finally, to ensure a seamless transition between the original and the recovered regions we apply image inpainting along a thin border. We demonstrate that our approach produces results that are in most cases superior in quality to those of previous shadow removal methods. We also show that it is possible to easily composite the extracted shadow onto a new background or modify its size and direction in the original image. 1. Introduction The removal of shadows from a single image is an inter- esting and important research problem. In computational photography one may wish to remove shadows due to aes- thetic reasons — for example, a shadow cast by the pho- tographer onto the scene, or shadows added due to the use of a flash. Shadows also often interfere with common com- puter vision tasks, such as segmentation, tracking, and object recognition [Ull96]. Thus, removing shadows as a prepro- cess to these tasks could result in improved performance. In addition, special effects often require removing objects, in which case their shadows must be removed as well. Finally, shadows provide important visual cues to our perception of shape, occlusion, contact, etc. Thus, being able to remove, add, and modify shadows in an image is an important image manipulation tool. The shadow removal process consists of two challenging subtasks: detecting the shadowed region and restoring the il- lumination in that region. The detection task involves some degree of image understanding in order to determine whether a pixel is dark due to a shadow or the reflectance at the corresponding scene point. Accomplishing this task requires making some assumptions about the shadowed surfaces in the scene and/or asking the user for some hints. The restora- tion task is also challenging, as it attempts to eliminate any perceivable differences between the originally lit and the re- stored parts of the image. In particular, it is difficult to avoid differences in local contrast and in the amount of noise be- tween the two regions. Another difficulty is presented by shadowed regions where the intensity of the shadow is non- uniform. In this paper we propose new techniques addressing both of the subtasks mentioned above. Our main technical con- tributions are in the illumination restoration stage, consist- ing of a novel pyramid-based restoration process that avoids loss of texture contrast in the recovered regions by applying an affine shadow recovery model at multiple scales. A more detailed overview of our approach and contributions follows the survey of related work below. 1.1. Related Work Shadow detection and removal are closely related to the re- covery of intrinsic images by separating an image to its re- flectance and illumination components [BT78]. It is possible to derive the reflectance component from a sequence of im- c  2008 The Author(s) Journal compilation c 2008 The Eurographics Association and Blackwell Publishing Ltd. Published by Blackwell Publishing, 9600 Garsington Road, Oxford OX4 2DQ, UK and 350 Main Street, Malden, MA 02148, USA.