ACM Reference Format Glencross, M., Ward, G., Jay, C., Liu, J., Melendez, F., Hubbold, R. 2008. A Perceptually Validated Model for Surface Depth Hallucination. ACM Trans. Graph. 27, 3, Article 59 (August 2008), 8 pages. DOI = 10.1145/1360612.1360658 http://doi.acm.org/10.1145/1360612.1360658. Copyright Notice Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or direct commercial advantage and that copies show this notice on the first page or initial screen of a display along with the full citation. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, to republish, to post on servers, to redistribute to lists, or to use any component of this work in other works requires prior specific permission and/or a fee. Permissions may be requested from Publications Dept., ACM, Inc., 2 Penn Plaza, Suite 701, New York, NY 10121-0701, fax +1 (212) 869-0481, or permissions@acm.org. © 2008 ACM 0730-0301/2008/03-ART59 $5.00 DOI 10.1145/1360612.1360658 http://doi.acm.org/10.1145/1360612.1360658 A Perceptually Validated Model for Surface Depth Hallucination Mashhuda Glencross ⇤ The University of Manchester Gregory J. Ward † Dolby Canada Caroline Jay ‡ The University of Manchester Jun Liu § The University of Manchester Francho Melendez ¶ The University of Manchester Roger Hubbold k The University of Manchester Figure 1: The left image is a photograph of a Mayan carving under diffuse lighting, which was combined with a similar flash photo to derive a height field and albedo map for this surface. The middle image uses the derived model to render the same view with novel lighting. The rendering on the right shows an oblique close-up with a second novel lighting condition and added specularity. Abstract Capturing detailed surface geometry currently requires specialized equipment such as laser range scanners, which despite their high ac- curacy, leave gaps in the surfaces that must be reconciled with pho- tographic capture for relighting applications. Using only a standard digital camera and a single view, we present a method for recover- ing models of predominantly diffuse textured surfaces that can be plausibly relit and viewed from any angle under any illumination. Our multiscale shape-from-shading technique uses diffuse-lit/flash- lit image pairs to produce an albedo map and textured height field. Using two lighting conditions enables us to subtract one from the other to estimate albedo. In the absence of a flash-lit image of a surface for which we already have a similar exemplar pair, we ap- proximate both albedo and diffuse shading images using histogram matching. Our depth estimation is based on local visibility. Unlike other depth-from-shading approaches, all operations are performed on the diffuse shading image in image space, and we impose no constant albedo restrictions. An experimental validation shows our method works for a broad range of textured surfaces, and viewers are frequently unable to identify our results as synthetic in a ran- domized presentation. Furthermore, in side-by-side comparisons, subjects found a rendering of our depth map equally plausible to one generated from a laser range scan. We see this method as a significant advance in acquiring surface detail for texturing using a standard digital camera, with applications in architecture, archaeo- logical reconstruction, games and special effects. CR Categories: I.3.7 [Computer Graphics]: Three-Dimensional Graphics and Realism—Color, shading, shadowing, and texture; Keywords: relighting, shape-from-shading, albedo estimation, textured surfaces, perception, computational photography ⇤ e-mail: mashhuda@manchester.ac.uk † e-mail: gward@dolby.com ‡ e-mail: caroline.jay@manchester.ac.uk § e-mail: jun.liu@manchester.ac.uk ¶ e-mail: francho.melendez@manchester.ac.uk k e-mail: roger.hubbold@manchester.ac.uk 1 Introduction Textured surfaces such as brick, stone, wood and many other build- ing materials have local variations in their surface meso-structure. Shading variations due to self-shadowing provide important per- ceptual cues necessary to convey a correct impression of shape. An interesting question, however, is how accurate does surface meso- structure need to be for shape and corresponding shading to ap- pear plausible? This is an important question as our objective is to produce synthetically relit results that are perceptually difficult to distinguish from photographs. (See Figure 1.) In this paper we show that an approximate representation of the real surface (depth + albedo map) may be used to relight predominantly diffuse textured surfaces in a visually plausible manner. To this aim, we introduce a practical method to recover approximate surface texture informa- tion from a single viewpoint. From a 2D picture, we infer surface depth where it is not fully divulged in the image. We call this depth hallucination. Representing surface detail is useful to increase the visual realism in a range of application areas, especially architectural reconstruc- tions. In particular, accurately assessing the effect of new build- ings on lighting requires modeling of gross 3D geometry, meso- structure, and albedo (equivalent to diffuse reflectance), so that simulations of appearance at different times of the day are possi- ble. Our method is aimed primarily at the materials recovery part of such architectural reconstructions. We aim to acquire surface meso-structure so that it may be combined with gross 3D geom- etry (obtained using another method) to convey the appearance of ACM Transactions on Graphics, Vol. 27, No. 3, Article 59, Publication date: August 2008.