Spatial Analysis of Discrete Plenoptic Sampling Andrew Lumsdaine a , Todor Georgiev b , and Georgi Chunev a a Indiana University, Bloomington, IN, USA; b Adobe Systems, San Jose, CA, USA ABSTRACT Plenoptic cameras are intended to fully capture the light rays in a scene. Using this information, optical elements can be applied to a scene computationally rather than physically—allowing an infinite variety of pictures to be rendered after the fact from the same plenoptic data. Practical plenoptic cameras necessarily capture discrete samples of the plenoptic func- tion, which together with the overall camera design, can constrain the variety and quality of rendered images. In this paper we specifically analyze the nature of the discrete data that plenoptic cameras capture, in a manner that unifies the traditional and focused plenoptic camera designs. We further present a resolution analysis for plenoptic cameras and develop design guidelines for maximizing resolution. A generalized rendering algorithm is presented that minimizes artifacts resulting from the lower resolution angular sampling that accompanies high-resolution spatial sampling. Experimental results using a real-time GPU implementation of our algorithms demonstrates the effectiveness of our approach. Keywords: plenoptic camera, rendering, discrete sampling 1. INTRODUCTION Integral photography has more than 100 years of history, starting with Ives 1 and Lippmann. 2 Lippmann motivated his work in this area by observing that even the “most perfect photographic print only shows one aspect of reality; it reduces to a single image fixed on a plane, similar to a drawing or a hand-drawn painting.” With integral photography, he attempted instead to render infinitely more—“the full variety offered by the direct observation of objects.” Unfortunately, because of inherent limitations of available technologies, the potential of integral photography was not realized. Integral photography has recently re-emerged with the introduction of the plenoptic camera, a device that captured the distribution of light rays in space (i.e., the 4D plenoptic function) in order to capture 3D imagery and solve computer vision problems. 3 The lightfield and lumigraph, introduced to the computer graphics community respectively in 4 and, 5 established a theoretical framework for analyzing the plenoptic function. A handheld plenoptic camera, along with new methods of processing, was introduced by Ng in. 6 Practical plenoptic cameras necessarily capture discrete samples of the plenoptic function, which together with the overall camera design, can limit the variety and quality of rendered images. The discretization comes from two sources: discrete microlenses in the microlens array and discrete pixels in the camera sensor. The effect of these sources is to impose a specific geometry on the overall sampling of the plenoptic function by plenoptic cameras. This geometry is subject to computational transforms applied when rendering images from the captured plenoptic function. Understanding the geometry of the captured lightfield is therefore fundamental to understanding the properties of the rendered image (most notably, resolution and depth of field). In this paper we analyze plenoptic cameras specifically from the point of view of discrete sampling. We use the optical properties of plenoptic cameras to derive the geometry of discrete plenoptic function capture. Based on this geometry, we derive expressions for expected resolution from a captured plenoptic function as well as necessary conditions in the optical design for overcoming the resolution limitations of the traditional plenoptic camera. 2. RELATED WORK The basic structure of the focused plenoptic camera can first be seen in the early work of Lippmann, 2 who considered the array of lenslets as an array of cameras focused on the photographed object. Lippmann proposed the idea of using a lenslet array to capture the radiance of a scene and produce what he called integral photographs. More recently, the basic principles of plenoptic cameras have been independently considered by a number of researchers including Ng, 7 Fife, 8 and Lumsdaine. 9