Focused plenoptic camera and rendering Todor Georgiev Adobe Systems Inc. 345 Parke Avenue San Jose, California 95110 E-mail: tgeorgie@adobe.com Andrew Lumsdaine Indiana University Computer Science Department 215 Lindley Hall Bloomington, Indiana 47405 Abstract. Plenoptic cameras, constructed with internal microlens arrays, capture both spatial and angular information, i.e., the full 4-D radiance, of a scene. The design of traditional plenoptic cameras assumes that each microlens image is completely defocused with respect to the image created by the main camera lens. As a result, only a single pixel in the final image is rendered from each microlens image, resulting in disappointingly low resolution. A recently devel- oped alternative approach based on the focused plenoptic camera uses the microlens array as an imaging system focused on the im- age plane of the main camera lens. The flexible spatioangular trade- off that becomes available with this design enables rendering of final images with significantly higher resolution than those from traditional plenoptic cameras. We analyze the focused plenoptic camera in optical phase space and present basic, blended, and depth-based rendering algorithms for producing high-quality, high-resolution im- ages. We also present our graphics-processing-unit-based imple- mentations of these algorithms, which are able to render full screen refocused images in real time. © 2010 SPIE and IS&T. DOI: 10.1117/1.3442712 1 Introduction Integral photography, introduced by Ives and Lippmann over 100 years ago 1,2 has more recently reemerged with the introduction of the plenoptic camera. Originally presented as a technique for capturing 3-D data and solving computer-vision problems, 3,4 the plenoptic camera was de- signed as a device for recording the distribution of light rays in space, i.e., the 4-D plenoptic function or radiance. The light field and lumigraph, introduced to the computer graphics community, respectively, in Refs. 5 and 6, estab- lished a framework for analyzing and processing these data. In 2005, Ng et al. 7 and Ng 8 improved the plenoptic camera and introduced new methods of digital processing, includ- ing refocusing. Because it captured the full 4-D radiance, Ng’s handheld plenoptic camera could produce effects well beyond the capabilities of traditional cameras. Image properties such as focus and depth of field could be adjusted after an image had been captured. Unfortunately, traditional plenoptic cameras suffer from a significant drawback; they render images at disappointingly low resolution. For example, im- ages rendered from Ng’s camera data have a final reso- lution of 300  300 pixels. A different approach, called “full-resolution lightfield rendering,” 9 can produce final images at much higher res- olution based on a modified plenoptic camera the “focused plenoptic camera” 10 . This modified camera is structurally different from the earlier plenoptic camera with respect to microlens placement and microlens focus. These structural differences in turn result in different assumptions about the sampling of the 4-D radiance. The traditional plenoptic camera focuses the main lens on the microlenses and fo- cuses the microlenses at infinity. The focused plenoptic camera instead focuses the main camera lens well in front of the microlenses and focuses the microlenses on the im- age formed inside the camera—i.e., each microlens forms a relay system with the main camera lens. This configuration produces a flexible trade-off in the sampling of spatial and angular dimensions and enables positional information in the radiance to be sampled more effectively. As a result, the focused plenoptic camera can produce images of much higher resolution than can traditional plenoptic cameras. Other radiance-capturing cameras similar to the focused plenoptic camera include the following: the microlens ap- proach of Lippmann, 2 the thin observation model by bound optics 11 TOMBO, the handheld plenoptic camera, 7 Fife et al.’s multiaperture image sensor architecture, 12,13 and the Panoptes sensor. 14 This paper presents the design and analysis of the fo- cused plenoptic camera and associated image rendering al- gorithms. In particular, it describes 1. the background of the plenoptic camera 2.0, includ- ing basic radiance theory and modeling, the plenoptic camera 1.0, and other radiance-capturing cameras 2. a complete development of the focused plenoptic camera, including derivation of its sampling in opti- cal phase space, basic rendering algorithms, and a detailed description of the hardware Paper 10039SSRR received Mar. 5, 2010; revised manuscript received Apr. 9, 2010; accepted for publication Apr. 20, 2010; published online Jun. 11, 2010. 1017-9909/2010/192/021106/11/$25.00 © 2010 SPIE and IS&T. Journal of Electronic Imaging 19(2), 021106 (Apr–Jun 2010) Journal of Electronic Imaging Apr–Jun 2010/Vol. 19(2) 021106-1 Downloaded from SPIE Digital Library on 29 Sep 2011 to 150.135.223.93. Terms of Use: http://spiedl.org/terms