Shape-from-Shifting: Uncalibrated Photometric Stereo with a Mobile Device Chia-Kai Yeh 1 , Fengqiang Li 1 , Gianluca Pastorelli 2 , Marc Walton 2 , Aggelos K. Katsaggelos 1 and Oliver Cossairt 1 1 Department of Electrical Engineering and Computer Science, Northwestern University, Evanston, IL, USA 2 Center for Scientific Studies in the Arts (NU-ACCESS), Northwestern University, Evanston, IL, USA Abstract—Surface shape scanning techniques, such as laser scanning and photometric stereo, are widespread analytical tools used in the field of cultural heritage. Compared to regular 2D RGB photos, 3D surface scans provide higher fidelity of an object’s surface shape which assist conservators, art historians, and archaeologists in understanding how these artworks and artifacts are made and to digitally document them for purposes of conservation. However, current state-of-the-art 3D surface scanning tools used in art conservation are often expensive and bulky- such as light dome structures that are often over 1 m in diameter. In this paper, we introduce mobile shape-from-shifting (SfS): a simple, low-cost and streamlined photometric stereo framework for scanning planar surfaces with a consumer mobile device coupled to a low-cost add-on component. Our free-form mobile SfS framework relaxes the rigorous hardware and other complex requirements inherent to conventional 3D scanning tools. This is achieved by taking a sequence of photos with the on-board camera and flash of a mobile device. The sequence of captures are used to reconstruct high quality normal maps using near- light photometric stereo algorithms, which are of comparable quality to conventional photometric stereo. We demonstrate 3D surface reconstructions with SfS on different materials and scales. Moreover, the mobile SfS technique can be used ”in the wild” so that 3D scans may be performed in their natural environment, eliminating the need for transport to a laboratory setting. With the elegant design and low cost, we believe our Mobile SfS can greatly benefit the conservation community by providing a user- friendly and cost-effective solution for 3D surface scanning. Index Terms—3D Surface Shape Reconstruction, Photomet- ric Stereo, Image-Based Modeling, Reflectance Transformation Imaging, Scale-invariant Feature Transform, Near-Light Position Calibration. I. I NTRODUCTION 3D imaging techniques have had an explosive growth in both industry and academic research during the last decade with a variety of applications, such as visual effects in movies and video games [3], computer-aided-design for rapid pro- totyping, quality inspection [4], and biological imaging [5], [6]. In the community of cultural heritage, 3D imaging has gained widespread popularity as a tool for documenting object condition [7]. 3D imaging methods can be loosely divided into two groups: passive and active 3D imaging. Passive based 3D imaging, such as photogrammetry relies on the reflected radiance from an object lit with ambient illumination to recon- struct the object’s 3D surface shape. Active 3D imaging, such as photometric stereo (PS) [8], uses a controlled light source, such as a flash light, to illuminate the object and recover the 3D surface shape. PS is a highly sensitive technique that is Image Registration based on SIFT Features Near-Light Photometric Stereo Surface Normal Map Photo shifting direction Images Capturing Fig. 1. Overview of Shape from Shifting: Our shape-from-shifting technique uses a mobile device camera to capture images around the object with the built-in flash used as a source of illumination. SIFT-based image registration renders the images to the same viewpoint but each with a unique illumination direction. The synthesized images are further processed by uncalibrated photometric stereo to acquire dense surface normal vector maps. capable of recovering 3D surface shape information on the scale of micrometers. For this reason, it has been widely used for the visualization of works of art and artifacts. While PS has been explored extensively, it still faces many fundamental challenges that limit its ease of use and has prevented its widespread adoption as a collection survey tool in the cultural heritage community. PS estimates the surface normal/shape from photos taken by a fixed position camera but with varying lighting position and direction. By modeling the measured image intensity as a func- tion of the incident lighting angle, one can recover the surface normal and material reflectance of each point on the object. The depth information of the object can then be recovered by integrating the reconstructed surface normals. The material’s reflectance can also be interactively manipulated by the users for the purposes of visualization and may be integrated into virtual reality (VR) head-mounted displays (HMDs) or aug- mented reality (AR) displays. Conventional PS reconstruction techniques use a far-light assumption, which assumes the position of the illumination light source is infinitely far away from the object. In practice, this assumption is frequently violated due to the space limitations of the acquisition setup. Without proper correction of the forward model, the violation