3D VIDEO AND FREE VIEWPOINT VIDEO – TECHNOLOGIES, APPLICATIONS AND
MPEG STANDARDS
Aljoscha Smolic, Karsten Mueller, Philipp Merkle, Christoph Fehn, Peter Kauff, Peter Eisert, and Thomas Wiegand
Fraunhofer Institute for Telecommunications/Heinrich-Hertz-Institut
Einsteinufer 37, 10587 Berlin, Germany
Aljoscha.Smolic@fraunhofer.hhi.de
ABSTRACT
An overview of 3D and free viewpoint video is given in this paper
with special focus on related standardization activities in MPEG.
Free viewpoint video allows the user to freely navigate within real
world visual scenes, as known from virtual worlds in computer
graphics. Examples are shown, highlighting standards conform
realization using MPEG-4. Then the principles of 3D video are
introduced providing the user with a 3D depth impression of the
observed scene. Example systems are described again focusing on
their realization based on MPEG-4. Finally multi-view video coding is
described as a key component for 3D and free viewpoint video
systems. The conclusion is that the necessary technology including
standard media formats for 3D and free viewpoint is available or will
be available in the near future, and that there is a clear demand from
industry and user side for such applications. 3DTV at home and free
viewpoint video on DVD will be available soon, and will create huge
new markets.
1. INTRODUCTION
Digital media have influenced and changed modern society over the
last 2 decades significantly. Media are more and more produced,
processed, stored, and transmitted in digital formats with digital
equipment. Applications, terminals and content are merging more and
more. We can watch TV with our mobile phones, surf the web with
the TV set, and modern home PCs are powerful multimedia
workstations capable of more or less everything. On a DVD we do
not buy only the movie with video and audio but also a vast amount
of supplementary information. New media formats integrate all types
of media, such as video, audio, computer graphics, text, images, etc.
into a single file or stream format. Some of these formats further
enable user interactivity with the content, i.e. the user can do
something with the media in addition to just passively consuming. An
important factor for this success story is the availability of
international standards for digital media formats. They provide
interoperability between different systems while still allowing for
competition among equipment and service providers. ISO MPEG is
one of the international standardization bodies that play an important
role in the digital media market.
Recent research and convergence of technologies from
computer graphics, computer vision, multimedia and related fields
enabled also the development of new types of media, such as 3D
video and free viewpoint video that expand the user’s sensation far
beyond what is offered by traditional media. The first offers a 3D
depth impression of the observed scenery (also referred to as stereo,
note that the term 3D may have different meanings in the context of
this paper), while the second allows for interactive selection of
viewpoint and direction within a certain operating range as known
from computer graphics. Some application scenarios may be based
on proprietary systems, as for instance already employed for (post-)
production of movies and TV content. On the other hand there are
also application scenarios that require interoperable systems, such as
3DTV broadcast or free viewpoint video on DVD. This may open
huge consumer markets for 3D displays, set-top boxes, media,
content, DVDs, HD-DVDs, BRDs, etc., along with the
corresponding equipment for production, transmission, etc.
Therefore the MPEG committee has been investigating the needs
for standardization in the area of 3D and free viewpoint video in a
group called 3DAV (for 3D audio-visual) [15] in recent years. Thus
far, the committee has provided an overview of relevant technologies
and has shown that a number of these technologies are already be
supported by existing standards such as MPEG-4. For the missing
elements, new standardization activities have been launched. Some
activities have already been completed, such as the new tools for the
efficient and high-quality representation of 3D video objects, which
have been adopted as part of the MPEG-4 Animation Framework
eXtension (AFX) specification [3]. Other more challenging activities
are still ongoing, such as the specification of a new standard for
multi-view video coding with associated camera parameters, which
will enable 3D and free viewpoint video systems as the final goal.
This paper gives an overview of the applications “free viewpoint
video” and “3D video” in sections 2 and 3, highlighting the related
standardization activities in MPEG. Section 4 addresses a related
upcoming standard for compression of multi-view video, and finally
section 5 concludes the paper and gives an outlook to the future in
this area.
2. FREE VIEWPOINT VIDEO
Free viewpoint video (FVV) offers the same functionality that is
known from 3D computer graphics. The user can choose an own
viewpoint and viewing direction within a visual scene, meaning
interactive free navigation. In contrast to pure computer graphics
applications, FVV targets real world scenes as captured by real
cameras. This is interesting for user applications (DVD of an
opera/concert where the user can freely chose the viewpoint) as well
as for (post-) production. Systems for the latter are already being
used (e.g. for sports, movies, EyeVision, Matrix-effects).
2161 1424403677/06/$20.00 ©2006 IEEE ICME 2006