Signal, Image and Video Processing https://doi.org/10.1007/s11760-018-01411-2 ORIGINAL PAPER Improved HEVC -domain rate control algorithm for HDR video Junaid Mir 1 · Dumidu S. Talagala 2 · Anil Fernando 2 · Syed Sameed Husain 2 Received: 16 March 2018 / Revised: 29 October 2018 / Accepted: 26 December 2018 © Springer-Verlag London Ltd., part of Springer Nature 2019 Abstract Widespread high dynamic range (HDR) video distribution via transmission and broadcast is imminent in the near future. However, the rate control (RC) algorithms in video coding standards, like high-efficiency video coding (HEVC), are optimized and designed for low dynamic range (LDR) content—making them inefficient in the rate-distortion (RD) sense, when applied to HDR video compression and distribution. In this paper, we propose a non-normative change to the HM16.2 HEVC λ- domain RC algorithm to achieve high-fidelity HDR video distribution. A new λ-QP relation is modeled for HDR content after assessing the suitable RD model. The results indicate that the proposed RC algorithm outperforms the default RC algorithm in HEVC, achieving on average performance gains of 1.40 dB, 0.685 units, and 0.0155 units in terms of PU-PSNR, HDR- VDP-2.2 Q factor, and HDR-VQM, respectively. Further, proposed method has bit estimation accuracy similar to the default HEVC RC algorithm. Also, the subjective evaluations corroborate the significance of the performance gained in the objective metrics. Keywords HDR · Rate control · HEVC · Rate-distortion · R-λ model 1 Introduction High dynamic range (HDR) content is the new evolution in digital multimedia seeking to offer the experience and qual- ity closer to the true-to-life visual sensation by providing the high-fidelity depiction of the real-world scene. With the rapid recent developments of HDR imaging and display technolo- gies, how to efficiently compress and distribute HDR content has been the main focus of discussion and investigation in relevant standardization bodies like Moving Picture Experts Group (MPEG) [1]. This is vital to conceive the HDR video distribution through transmission or broadcast and for gen- eral acceptance of HDR technology by masses. Rate control (RC) algorithm ensures the efficient utiliza- tion of the bandwidth-constrained channel while maintaining the best possible quality. It plays a significant role in high-fidelity content distribution through communication channels and is therefore generally part of all video cod- ing standards. The RC algorithm in the state-of-the-art video B Junaid Mir junaidmir14@gmail.com 1 Department of Electrical Engineering, University of Engineering and Technology (UET) Taxila, Taxila, Pakistan 2 Center for Vision Speech and Signal Processing (CVSSP), University of Surrey, Guildford, UK coding standard HEVC is based on the λ-domain model [2]. In contrast to the previously adopted Q-domain [3] and ρ - domain models [4], λ-domain model is particularly adopted in the HEVC due to its increased RD performance and suit- ability for flexible coding structures introduced in HEVC. The existing HEVC λ-domain RC algorithm with high bit- depth support can be used for the HDR video compression and distribution as HDR video can be represented effectively in high bit-depth perceptually uniform integer-based color space (10- to 16- bits per color channel) [5]. However, one needs to consider that video coding standards and tools are generally optimized and designed keeping in view the char- acteristics of the low dynamic range (LDR) content [6,7]. For example, the foundation of HEVC λ-domain RC algo- rithm is based on the finding of robust relation between the bitrate R and λ which is derived from the RD relation char- acterized by the Hyperbolic function [2]. However, as the perceivability of HDR distortion is different due to the bright visual stimuli, characterization of the RD relation for HDR video by the Hyperbolic function needs to be assessed. Sim- ilarly, the relation [8] between λ and quantization parameter (QP) used in the HEVC λ-domain RC algorithm is currently modeled for the LDR content. This relation plays a vital role in the coding performance of RC as the resulting QP value, computed from the chosen λ, sets the quantization step and 123