978-1-4244-1724-7/08/$25.00 ©2008 IEEE ICALIP2008 1492 Complexity Reduced Mode Selection of H.264/AVC Intra Coding Mohammed Golam Sarwer 1,2 , Lai-Man Po 1 , Jonathan Wu 2 1 Department of Electronic Engineering City University of Hong Kong Kowloon, Hong Kong 2 Department of Electrical and Computer Engineering University of Windsor Windsor, ON, Canada N9B3P4 sarwer@uwindsor.ca,eelmpo@cityu.edu.hk , jwu@uwindsor.ca Abstract One of the new features in the H.264/AVC encoder is the use of Lagrangian Rate-Distortion Optimization (RDO) method during mode decision at the macroblock level. The RDO technique has been employed in H.264/AVC for intra prediction mode selection to achieve better coding efficiency. But the computation complexity of mode decision algorithm is extremely high. To reduce the complexity of mode decision, we propose an efficient and fast 4x4 intra prediction mode selection scheme. The proposed method reduces the candidate of the prediction modes based on the Sum of Absolute Transformed Difference (SATD) between the original block and the intra predicted block. Rank of each mode is obtained based on the SATD value. The candidate modes are further reduced by using the combination of rank and most probable mode. Simulation results demonstrate that the proposed mode decision method reduces about 95 % of mode decision time of intra coding with ignorable degradation of coding performance. 1. Introduction H.264/AVC is the newest international video coding standard [1]. H.264/AVC offers very high compression ratios as compared to previous standards. To achieve the highest coding efficiency, H.264/AVC uses rate distortion optimization (RDO) technique. However, the computation load of H.264/AVC increase drastically compared to any previous standards. To reduce the complexity of intra 4x4 mode decision, H.264/AVC reference software suggested [2] sum of absolute difference (SAD) and sum of absolute transform difference (SATD) based cost functions. These two cost functions reduce computation significantly but performance of rate-distortion (RD) characteristics is not good enough. Several fast intra mode-decision approaches proposed in [3-6] focused on how to eliminate unnecessary modes. Meng et al. [3] selected optimal mode by computing a partial cost for down-sampled pixels instead of 4x4 block. To find the best intra mode, the information of the edge map of the whole frame is proposed in [5]. A fast intra mode decision method based on sum of absolute transform differences (SATD) between actual and predicted block is presented in [6] but the complexity reduction is not high. In this paper, we propose a fast mode decision scheme for intra 4x4 prediction in H.264/AVC based on SATD. Natural video sequences are highly spatially correlated. So the probability of mode of upper or left block to be the best mode of current block is high. Based on these observations, the proposed method reduces the number of candidate modes to either 1 or 2. 2. Overview of 4x4 intra prediction 2.1 Cost function for Intra 4x4 mode decision To take the full advantages of all modes, the H.264/AVC encoder can determine the mode that meets the best RD tradeoff using RDO mode decision scheme. The best mode is the one having minimum rate-distortion cost and this cost is expressed as R SSD J RD λ + = (1) where, SSD is the sum of squared difference between the original blocks and the reconstructed block , R is the true bits needed to encode the block and λ is an exponential function of the quantization parameter (QP). To reduce the computational complexity, H.264/AVC reference software provides an SATD- based cost function: P SATD J SATD 4 1 ⋅ + = λ (2)