Low-Bitrate Video Quality Enhancement by Frame Rate Up- Conversion and Adaptive Frame Encoding Ya-Ting YANG 1 , Yi-Shin TUNG 2 , Ja-Ling WU 1,2 and Chung-Yi WANG 1 1 Department of Computer Science and Information Engineering, National Taiwan University 2 The Graduate Institute of Networking and Multimedia, National Taiwan University {ytyang, tung, wjl, chunye}@cmlab.csie.ntu.edu.tw Abstract. Frame rate up-conversion (FRUC) is a useful technique for a lot of practical applications, such as display format conversion, low bitrate video coding and slow motion playback. Unlike traditional approaches, such as frame repetition or linear frame interpolation, motion-compensated frame interpolation (MCFI) technique which takes block motion into account is regarded as a more efficient scheme. By considering the deficiencies in previous works, new criteria and coding schemes for improving motion derivation and interpolation processes are suggested. We then integrate the proposed MCFI scheme into the decoding process of the latest coding standard, H.264/AVC. In addition, adaptive frame skip is fulfilled at the encoder side to maximize the power of MCFI in video coding applications. As a result, the encoder can adopt the MCFI dynamically and can decide whether the input frame should be coded or dropped then interpolated. Experimental results show that our proposal indeed enhances the overall quality, both subjectively and objectively, especially for the low bitrate video coding. Keywords: Frame Rate Up-Conversion, Motion Compensated Frame Interpolation, Adaptive Frame Skip 1 Introduction The popularity of advanced television and multimedia information systems has caused a rapid increase in the number of video sources and variety of display formats. This has resulted in a demand for converting between various formats efficiently. In contrast with super-resolution video reconstruction [1], frame rate up conversion (FRUC), as implied by the name, is a process to convert the video frame rate from a lower number to a higher one. When a video sequence is encoded to a certain compression ratio, frame interpolation technique (FIT) is always used as a post-processing tool to reconstruct the skipped frames. FIT reduces the temporal jerkiness by representing video at any desired frame rate (even to the full frame rate) on the basis of interpolation technique. The development of FRUC potentiates a lot of video applications. The most practical one, probably, is to enhance the reconstructed quality of a low bitrate coded video. For example, in the video conferencing applications, it is inevitable for an encoder to have a number of frames skipped. A temporal interpolation is then helpful to smooth over those discontinuities. Besides these applications, FRUC can also benefit slow-motion playback by synthesizing those inexistent intermediate frames for smoothing slow motion playback. Other well-known applications include PAL-NTSC conversion, and the video editing. FRUC may also impact on the rate allocation policy of a scalable video coding scheme. The conventional FRUC approaches (such as frame repetition and linear frame interpolation) did not take motion information into account. As long as the video sequence has large or complex motion, those approaches may fail, and annoying artifacts such as motion jerkiness or image blur may appear.