VIEWPORT-AWARE ADAPTIVE 360° VIDEO STREAMING USING TILES FOR VIRTUAL REALITY Cagri Ozcinar, Ana De Abreu, and Aljosa Smolic Trinity College Dublin (TCD), Dublin 2, Ireland. ABSTRACT 360° video is attracting an increasing amount of atten- tion in the context of Virtual Reality (VR). Owing to its very high-resolution requirements, existing professional stream- ing services for 360° video suffer from severe drawbacks. This paper introduces a novel end-to-end streaming system from encoding to displaying, to transmit 8K resolution 360° video and to provide an enhanced VR experience using Head Mounted Displays (HMDs). The main contributions of the proposed system are about tiling, integration of the MPEG- Dynamic Adaptive Streaming over HTTP (DASH) standard, and viewport-aware bitrate level selection. Tiling and adap- tive streaming enable the proposed system to deliver very high-resolution 360° video at good visual quality. Further, the proposed viewport-aware bitrate assignment selects an optimum DASH representation for each tile in a viewport- aware manner. The quality performance of the proposed system is verified in simulations with varying network band- width using realistic view trajectories recorded from user experiments. Our results show that the proposed streaming system compares favorably compared to existing methods in terms of PSNR and SSIM inside the viewport. Index Terms— 360° video, virtual reality, tiling, DASH, viewport-aware 1. INTRODUCTION During the last years, significant achievements have been made regarding the rendering capacity and quality of Head Mounted Display (HMD) systems [1]. Modern HMDs can render 360° video at a sufficiently high frame-rate and reso- lution, allowing the viewer to be immersed in the VR envi- ronment. Although numerous professional VR video services have emerged for streaming the 360° video, e.g., YouTube 360 [2], they suffer from severe drawbacks. More clearly, each video frame, containing a 360° Field of View (FOV), is encoded and transmitted regardless of the FOV of the HMD. In fact, the existing HMDs have a viewable FOV that ranges from 96° to 110° [3], meaning they use only around one fifth of the transmitted data [4]. The area of the 360° video frame dis- played by the HMD at a given time is known as the viewport. Hence, the perceptual quality of such video mainly depends on the viewport quality. In existing professional services, re- gions outside the viewport waste a considerable proportion of the bandwidth. This unnecessarily utilized bandwidth re- sults in overall low-quality video streaming to comply with the present Internet and decoding limitations. Considering its data-intensive representation and the best- effort nature of the Internet, 360° video streaming requires a bitrate adaptive solution to offer an enhanced VR experi- ence. To this end, MPEG-Dynamic Adaptive Streaming over HTTP (DASH) [5], which is an international standard for adaptive streaming, is a key enabler in achieving a smooth VR video playback. The main aim of DASH is to provide a high- quality streaming experience based on the client bandwidth. In DASH, video streams are requested using a manifest file, Media Presentation Description (MPD), which contains a set of bitrate representations. The significant requirements regarding resolution to en- sure high-quality VR experience [3, 6] can be managed with tiling and viewport-aware solutions using DASH. For exam- ple, tiling can generate self-decodable regions, i.e., tiles, by spatially dividing the frames. In addition, tiling can consider the importance of regions [7] and can enable parallel down- loading [8] and decoding [9, 10] features. As the HMDs use only the viewport out of the captured 360° video, the visual quality of the viewport can be enhanced in a viewport-aware manner. This paper focuses on adaptive 360° video streaming and provides improved viewport quality compared to existing pro- fessional services. A VR video streaming system, from en- coding to displaying, is designed to transmit 8K 360° video and to offer an enhanced VR experience. The main contri- bution of this work is an end-to-end streaming system im- plementation that contains tiling, a novel extension of the MPD, and DASH bitrate level selection in a viewport-aware manner. Experimental results showed that the proposed sys- tem demonstrates significant quality enhancements compared with the streaming approach that is currently used by profes- sional VR services [2, 11] in this area. Our work concentrates on adaptive distribution of bitrate and quality over the 360° video in contrast to existing professional services. The sys- tem uses the tiling concept to divide each video frame into tiles to encode, transmit, and decode them effectively. To fa- cilitate tiled 360° video streaming, we extended the concept arXiv:1711.02386v1 [cs.MM] 7 Nov 2017