Synthesis of a high-resolution 3D-stereoscopic image pair from a high-resolution monoscopic image and a low-resolution depth map Kyung-tae Kim a,* , Mel Siegel a , Jung-Young Son b a The Robotics Institute School of Computer Science, Carnegie Mellon University, Pittsburgh PA 15213 b Korea Institute of Science and Technology (KIST), Seoul, Korea ABSTRACT We describe a new low-level scheme to achieve high definition 3D-stereoscopy within the bandwidth of the monoscopic HDTV infrastructure. Our method uses a studio quality monoscopic high resolution color camera to generate a transmitted “main stream” view, and a flanking 3D-stereoscopic pair of low cost, low resolution monochrome camera “outriggers” to generate a depth map of the scene. The depth map is deeply compressed and transmitted as a low bandwidth “auxiliary stream”. The two streams are recombined at the receiver to generate a 3D-stereoscopic pair of high resolution color views from the perspectives of the original outriggers. Alternatively, views from two arbitrary perspectives between (and, to a limited extent, beyond) the low resolution monoscopic camera positions can be synthesized to accommodate individual viewer preferences. We describe our algorithms, and the design and outcome of initial experiments. The experiments begin with three NTSC color images, degrade the outer pair to low resolution monochrome, and compare the results of coding and reconstruction to the originals. KEYWORDS: stereoscopy, stereo cameras, 3D-TV, disparity, occlusion, view interpolation 1. INTRODUCTION 1 With the arrival of digital HDTV equipment and systems, 3D-stereoscopic displays for teleoperation, videoconferencing, entertainment, etc., will becoming increasingly straightforward to implement within the monoscopic infrastructure. The value of the added realism must outweigh any required increases in processing and system complexity, and the 3D-stereoscopic information must be comfortable to view. Practically speaking, the primary costs are the very wide bandwidth that is needed to transmit the second image stream, the capital cost the additional high resolution cameras, and the functional costs of using 3D- stereoscopic camera rigs that are at least as cumbersome as two separate cameras that have to be maintained in precise mechanical, optical, and electronic alignment. The wide bandwidth requirement is the most fundamental obstacle, inasmuch as it is likely that in the foreseeable future 3D-TV will have to operate within the confines of the next few generations of the monoscopic TV infrastructure. To transmit 3D-stereoscopic imagery in this environment requires accommodating to the bandwidth allocated. In NTSC and other current analog systems this is typically accomplished by discarding half the lines or half the columns in each view. Many workers in the video coding and compression field have proposed schemes for mixed resolution coding, joint coding, etc., which trade off, in fixed bandwidth, just a little resolution in exchange for full stereopsis. The unifying feature of all these schemes is that the redundancy between perspective views is exploited so as to devote almost all of the available bandwidth to that which is common to both views, and only the remaining small fraction of the total bandwidth to that which is different in the two views. In this paper we describe a new scheme, and report on initial experiments with it, to implement this philosophy at the lowest hardware level. We use a monoscopic high resolution color camera to generate the transmitted “main stream” view, and a flanking 3D-stereoscopic pair of low resolution monochrome “outrigger” cameras to generate the transmitted “auxiliary stream”. The outriggers are used, at the source end, to compute the disparity map of the scene captured by the central high resolution color camera; the auxiliary stream is essentially this disparity map. The main stream and the auxiliary stream are combined at the receiver to generate a 3D-stereoscopic pair of high resolution color views from the perspectives of the original low resolution monochromatic outrigger cameras. Alternatively, views from two arbitrary perspectives between (and, to a * Further author information -- K. T. Kim: on sabbatical from Hannam University, Taejon, Korea., email: ktkim@eve.hannam.ac.kr. Mel Siegel (correspondence): mws+@cmu.edu, http://www.cs.cmu.edu/~mws, voice 412 268 8802, fax 412 268 5569.