Spatio-temporal Reflectance Sharing for Relightable 3D Video Naveed Ahmed, Christian Theobalt, and Hans-Peter Seidel MPI Informatik, Saarbr¨ ucken, Germany, [nahmed,theobalt,hpseidel]@mpi-inf.mpg.de, WWW home page: http://www.mpi-inf.mpg.de/ Abstract. In our previous work [21], we have shown that by means of a model-based approach, relightable free-viewpoint videos of human actors can be reconstructed from only a handful of multi-view video streams recorded under calibrated illumination. To achieve this purpose, we employ a marker-free motion capture approach to measure dynamic human scene geometry. Reflectance samples for each surface point are captured by exploiting the fact that, due to the person’s motion, each surface location is, over time, exposed to the acquisition sensors under varying orientations. Although this is the first setup of its kind to mea- sure surface reflectance from footage of arbitrary human performances, our approach may lead to a biased sampling of surface reflectance since each surface point is only seen under a limited number of half-vector directions. We thus propose in this paper a novel algorithm that reduces the bias in BRDF estimates of a single surface point by cleverly taking into account reflectance samples from other surface locations made of similar material. We demonstrate the improvements achieved with this spatio-temporal reflectance sharing approach both visually and quanti- tatively. 1 Introduction The capturing of relightable dynamic scene descriptions of real-world events re- quires the proper solution to many different inverse problems. First, the dynamic shape and motion of the objects in the scene have to be captured from multi- view video. Second, the dynamic reflectance properties of the visible surfaces need to be estimated. Due to the inherent computational complexity, it has not been possible yet to solve all these problems for general scenes. However, in pre- vious work [21] we have demonstrated that the commitment to an adaptable a priori shape model enables us to reconstruct relightable 3D videos of one specific type of scene, namely of human actors. By means of a marker-free optical mo- tion capture algorithm, it becomes possible to measure both the shape and the motion of a person from multiple synchronized video streams [2]. If the video footage has, in addition, been captured under calibrated lighting conditions, the video frames showing the moving person not only represent texture samples, but actually reflectance samples. Since a description of time-varying scene geometry