Energy-Aware Video Coding of Multiple Views via Workload Balancing Domenic Forte, Ankur Srivastava University of Maryland, College Park, USA {dforte, ankurs}@umd.edu Abstract—Video coding and compression reduce the stor- age space and/or the bandwidth required to transmit video. Multiview video systems make use of more than one camera to describe a scene and therefore demand more computing resources and energy consumption to compress and transmit video data. In this paper, we present an energy management framework for multiview systems that combines the two ma- jor video coding and compression paradigms: Predictive and Distributed Video Coding (PVC and DVC). The paradigms generally assume that the decoder (in PVC) and the encoder (in DVC) are resource constrained devices. However, there are a growing number of applications where resources are constrained at both encoder and decoder. Such scenarios cannot be handled due to the imbalance in PVC/DVC and demand a more flexible paradigm. In the proposed framework, video coding that combines PVC and DVC is used to balance multiview video coding workload between the video encoder and decoder in a way that maximizes system lifetime. Simulation results show that the proposed method can obtain a lifetime percentage increase of 58% and 30% on average and maintain video quality when compared to strict PVC and DVC systems respectively. I. I NTRODUCTION Video-based applications require coding and compres- sion algorithms, such as MPEG, to transmit and/or store video data. However, video coding algorithms are com- putationally intensive and energy demanding, even after being fully optimized with existing software and hardware minimization techniques [1]. Thus, for devices with se- vere energy constraints or limited computing capabilities (eg. smart phones), coding video streams is challenging. Furthermore, emerging applications in multiview video systems such as free viewpoint television make use of more than one camera to describe a scene. The multiview scenario involves more video coding and, hence, even greater energy requirements. While energy minimization and video coding under energy constraints has been well studied in the literature for single camera systems (eg. [2], [1]), our survey only yielded [3] for multi-camera systems. In this paper, we examine a multiview coding system consisting of one encoder (sender) and one decoder (re- ceiver). We investigate ways of improving energy utiliza- tion of both sides of the system by combining the two major video coding paradigms discussed in the literature. The first paradigm is Predictive Video Coding (PVC) where the video encoder (sender) exploits temporal corre- lation between video frames to achieve high compression. PVC was designed to benefit applications with down-link models where a sender encodes a video signal once and that signal is decoded by many receivers. The second paradigm is called Distributed Video Coding (DVC) which exploits correlations at the decoder allowing for a low complexity encoder [4]. DVC benefits applications with an up-link model where one or more resource constrained senders transmit to one receiving sink. In the conventional paradigms, only one side of the system (encoder or decoder) is considered resource con- strained. In PVC, the constrained side is the decoder who performs a limited amount of work. In DVC, the encoder is constrained. Typical energy minimization techniques for PVC and DVC operate by reducing workload at the constrained side to produce a better energy footprint. However, such approaches are less effective when both encoder and decoder are energy constrained (such as for video transfer between smart phones). Our recent work [5] discussed a PVC/DVC hybrid codec for single camera (monoview) systems, which shifted workload between en- coder and decoder through a control parameter. Workload was dynamically adapted based on current energy/thermal conditions of the encoder and decoder. A. Contributions Although the framework proposed in [5] worked well for a monoview coding scenario, there are more correlation opportunities present in multiview scenarios which are not fully exploited. In this paper, we extend our original frame- work from [5] to the multiview case. In multiview coding, there are three ways that correlation between video frames is exploited for higher compression: (1) between frames of different cameras, (2) between frames of the same camera, and (3) between frames of both different cameras and the same camera. In our new framework, workload is shifted between the encoder and decoder for each of the above three cases. The end goal is to maximize the entire coding system’s lifetime (i.e the length of time before a limited energy supply is exhausted at the either the encoder or decoder sides). By shifting workload to the side with more remaining energy, we can increase the system’s lifetime. To this end, we propose (i) two types of control parameters for shifting workload; (ii) a controller which determines up to two control parameter values per video frame to maximize system lifetime. Simulation results show that the proposed method can obtain a lifetime percentage increase of 58% and 30% on average and maintain video quality when compared to strict PVC and DVC multiview systems respectively. Results also support our claim that the monoview codec from [5] yields unsatisfactory system lifetime results when applied to multiview coding. The rest of the paper is organized as follows. In the next section, we describe the background for monoview and multiview video coding. In Section III, we discuss our proposed multiview codec and control parameters which shift workload between encoder and decoder. Section IV contains our objective formulation, optimization frame- work, and energy models. Section V discusses how our controller determines optimal control parameter values and ways to reduce its complexity. Simulation results are presented in Section VI. 2011 NASA/ESA Conference on Adaptive Hardware and Systems (AHS-2011) 978-1-4577-0599-1/11/$26.00 ©2011 IEEE 295