RAW PLENOPTIC VIDEO CODING UNDER HEXAGONAL LATTICE RESOLUTION OF MOTION VECTORS Thuc Nguyen Huu, Vinh Duong Van, Jonghoon Yim, and Byeungwoo Jeon Department of Electrical and Computer Engineering, Sungkyunkwan University, Korea ABSTRACT In raw plenoptic video, the optimal motion searching points mostly follow the hexagonal structure of micro-images. Based on this understanding, we propose a new motion vector resolution, namely the hexagonal lattice (HL) resolution which reflects micro-image structure. The HL resolution can be efficiently represented by HL basis. A study in this paper shows that motion vectors are highly concentrated at hexagonal lattice points, leading to use of the proposed resolution in the context of video compression. In this regard, we demonstrate the compression benefit brought by estimating motion vectors at HL resolution in the VVC codec. Index Terms— plenoptic video, lenslet video, motion vector resolution, hexagonal lattice resolution, video coding. 1. INTRODUCTION 1 Plenoptic camera [1] [2] records the 4D plenoptic function [3] (or 4D light field [4]) by inserting an additional microlens array (MLA) between main lens and image sensor as in Fig. 1. The sensor output of plenoptic camera is known as the raw plenoptic image since it yet requires additional pre- processing steps to generate final plenoptic function. The plenoptic image consists of many micro-images (drawn as circles in Fig. 1) which are distributed following a particular structure. Here, the structure of micro-images depends on MLA layout which can be either in rectangular or hexagonal. To increase the fill-factor of a sensor, the MLA often follows the hexagonal layout, leading to hexagonal structure of micro-images. The raw plenoptic data, especially for raw plenoptic video, calls for compression due to its extremely high spatial resolution (it could be up to 7728 x 5368 for sensor output of Lytro camera). In this regard, motion estimation and compensation of plenoptic video are essential enabling techniques to effectively reduce the temporal redundancy. Unlike conventional video, plenoptic video has a unique characteristic to utilize in motion estimation, i.e., its optimum This research was supported by Basic Science Research Program through the National Research Foundation of Korea (NRF) funded by the Ministry of Science and ICT (NRF-2020R1A2C2007673). searching points mostly (around 80%) follow the micro- images structure [5][6]. This interesting property is illustrated in Fig. 2(b) which plots the motion vectors (MVs) of raw plenoptic video estimated by full search. The observation that motion vectors are, in general, sparsely located in plenoptic video motivated development of fast search methods [5][6] which can speed up the motion search while maintaining the compression performance. Noting that previous works [5][6] only paid attention to fast motion search algorithms, we suggest that the special motion property is not fully utilized yet, and propose a new motion vector resolution, namely the hexagonal lattice (HL) resolution. In this paper, we study using the HL resolution of the plenoptic video since it may work efficiently with the conventional MV resolutions in video coding standards (e.g., H.264/AVC, H.265/HEVC, H.266/VVC, etc.) considering its better rate distortion performance than the conventional MV resolutions for most cases. We focus on the sensor output of Lytro camera [1] (standard plenoptic camera) and the research for Raytrix camera [2] is left for future work. The paper is organized as follows. In the next section, we present fundamental background of motion vector and motion vector resolution in video coding standards. Following, in Section 3, we propose the HL resolution which is a new MV resolution for raw plenoptic video. A comprehensive study of HL resolution and its usage are also presented. In Section 4, we give experimental results and next we conclude the paper. Fig. 1. Optical structure of plenoptic camera and its sensor output Image sensor Optical axis MLA Main lens micro- image Raw plenoptic data (Lenslet image) object h D v D 1621 978-1-6654-0540-9/22/$31.00 ©2022 IEEE ICASSP 2022 ICASSP 2022 - 2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) | 978-1-6654-0540-9/22/$31.00 ©2022 IEEE | DOI: 10.1109/ICASSP43922.2022.9747504