Real Time Turbulent Video Super-Resolution Using MPEG 4 Barak Fishbain * , Leonid P. Yaroslavsky, Ianir A. Ideses Department of Electrical Engineering - Physical Electronics, the Iby and Aladar Fleischman Faculty of Engineering, Tel-Aviv University, Tel Aviv 69978, Israel ABSTRACT It has been shown that one can make use of local instabilities in turbulent video frames to enhance image resolution beyond the limit defined by the image sampling rate. This paper outlines a real-time solution for the implementation of super-resolution algorithm on MPEG-4 platforms. The MPEG-4 video compression standard offer, in real-time, several features, such as motion extraction with quarter pixel accuracy, scene segmentation to video object planes, global motion compensation and de-blocking and de-ringing filters, which can be incorporated into the super-resolution process to produce enhanced visual output. Experimental verification on real-life videos is also provided. Keywords: Turbulent video, Real-Time, Super-Resolution, MPEG-4 1. INTRODUCTION In long distance observation systems, images and video are frequently damaged by atmospheric turbulence, which causes spatially and temporally chaotic fluctuations in the index of refraction of the atmosphere [1] and results in chaotic, spatial and temporal geometrical distortions of neighborhoods of all pixels. This geometrical instability of image frames heavily worsens the quality of videos and hampers their visual analysis. To make visual analysis possible, it is required first of all to stabilize images of stable scenes while preserving real motion of moving objects that might be present in the scene. Methods of generating stabilized videos from turbulent videos, including real time ones, were reported in Refs.[2,3,4,5]. In [6], the idea was advanced of making a profit from atmosphere turbulence-induced image geometrical spatial/temporal degradations to compensate image sampling artifacts and generate stabilized images of the stable scene with higher resolution than that defined by the camera sampling grid. Recovering the high-frequency information is possible as multiple low-resolution (LR) observations may provide additional information about the high-frequency data. This information is introduced through sub-pixel displacements in the sampling grid, which enables the recovery of resolution. To this end, it is required that the LR observations contain different but related views of the scene. Super-resolution principles are detailed in the works of S. Srinivasan and R. Chellappa [7], Galatsanos, Wernick and Katsaggelos [8], R.R. Schultz [9] and T. J. Schultz [10]. Several researchers treat the problem of high resolution image recovery by designing an efficient multi-frame filtering algorithms, that account for both intra-frame (spatial) and inter-frame (temporal) correlations, for restoring image sequences that are degraded both by blur and noise [11, 12]. Others have formulated solutions to global motion problems, usually from an application perspective [13, 14, 15, 16, 17, 18, 19, 20, 21, 22]. With that, all of the above methods take solely the raw sequences' frames as input, without taking into consideration the presence of moving objects and super-resolution is achieved through computational complex algorithms. Nowadays most digital footage data is transmitted and stored using the International Telecommunication Union (ITU) and Moving Picture Experts Group (MPEG) coding standards [23,24]. Super-resolution techniques that have been designed for raw, uncompressed, video may not be effective when applied to compressed video because they do not incorporate the compression process into their models. This has raised the need for super-resolution techniques which utilize the standards' various features [25,26,27,28]. However, the suggested methods do not comply with real-time constrains. A real-time super-resolution method, over video encoder hardware, was presented in [29]. Yet, the suggested method utilized proprietary compression standard and exploited only the sequence's motion field. Using a common compression standard, such as ITU H.264 or MPEG-4 presents several benefits, such as real-time compatibility and broad availability of software and hardware implementations. This paper describes a practical super-resolution scheme which utilizes MPEG-4 features for producing, in real-time, good quality higher-resolution videos from low-resolution turbulent degraded video streams with discrimination of turbulent from real motion which is caused by moving objects or global camera translations. * barak@eng.tau.ac.il;phone 972-3-640-8014; fax 972-3-641-0189 Proceedings Real-Time Image Processing 2008 conference, SPIE Paper Number: 6811-5