BIT RATE ESTIMATION FOR COST FUNCTION OF 4x4 INTRA MODE DECISION OF H.264/AVC Mohammed Golam Sarwer, and Lai Man Po Department of Electronic Engineering, City University of Hong Kong Kowloon , Hong Kong SAR, China sarwer@ee.cityu.edu.hk , eelmpo@cityu.edu.hk ABSTRACT H.264/AVC is a newest international video coding standard that can achieve considerably higher coding efficiency than previous standards. This comes at the cost of the complex mode decision procedure using the rate-distortion optimization, which makes real-time encoding difficult. To reduce the complexity of rate-distortion cost, we propose a bit rate estimation technique to avoid the entropy coding method during mode decision of intra prediction. The estimation method is based on the properties of context- based variable length coding (CAVLC). Simulation results demonstrate that the proposed estimation method achieves up to 53 % reduced encoding time of intra coding with ignorable degradation of coding performance. 1. INTRODUCTION The appearance and development of various new multimedia services have need for higher coding efficiency. The ITU-T/ISO/IEC Joint Video Team established the newest video coding standard known as H.264/AVC [1]. H.264/AVC offers a significant performance improvement over previous video coding standards such as H.263++ and MPEG-4 [2,3]. New and advanced techniques are introduced in this new standard, such as intra prediction for I-frame encoding, multi-frames inter prediction, small block-size transform coding, context-adaptive arithmetic entropy coding, de-blocking filtering, etc. These advanced techniques make this new standard provides approximately 50% bit rate saving for equivalent perceptual quality relative to the performance of prior standards. To achieve the highest coding efficiency, H.264/AVC uses rate-distortion optimization (RDO) technique to get the best coding result in terms of maximizing coding quality and minimizing bit rates. This means that the encoder has to code the video by exhaustively trying all the mode combinations including the different intra and inter prediction modes. Therefore, the complexity and computation load of video coding in H.264/AVC increase drastically compared to any previous standards. To reduce computational complexity of H.264/AVC, a number of efforts have been made to explore the fast algorithm in motion estimation, intra mode prediction and inter mode prediction for H.264/AVC video coding [4-6]. In 4x4 intra mode decision, 9 prediction modes are used. To reduce the complexity, H.264/AVC reference software suggested [7] sum of absolute difference (SAD) and sum of absolute transform difference (SATD) based cost functions. These two cost functions reduce computation significantly but performance of rate-distortion characteristics is not good enough. To improve the rate-distortion performance, a new cost function for intra 4x4 mode decision is proposed in [8]. In this cost function, sum of absolute integer transform difference (SAITD) is used in distortion part and a rate prediction algorithm is used in rate part. The major drawback of this cost is that the bit estimation method can not give the very good estimation. In this paper, we propose a shortcut way to get the number of entropy coded bits as soon as the transform coefficients are quantized. A method for estimation of rate for cost function of 4x4 intra mode decision is proposed. This method is based on the properties of context-based variable length coding (CAVLC) and observation of VLC tables. 2. COST FUNCTION FOR INTRA 4X4 MODE DECISION OF H.264/AVC To get a richer set of intra-prediction patterns, H.264/AVC offers 9 prediction modes for 4x4 luma blocks and 4 prediction modes for 16x16 luma blocks. The best mode is the one having minimum rate-distortion cost and this cost is expressed as R SSD J RD ⋅ + = λ (1) where, the SSD is the sum of squared difference between the original block and the reconstructed block, and R is the true bits needed to encode the block and Ȝ is an exponential function of the quantization parameter (QP). In order to compute RD cost for each mode, same operation of forward and inverse transform/quantization and variable length coding is repetitively performed. All of these 1579 1-4244-1017-7/07/$25.00 ©2007 IEEE ICME 2007