A ROBUST BLOCK BASED IMAGE/VIDEO REGISTRATION APPROACH FOR MOBILE IMAGING DEVICES 1 A Robust Block Based Image/Video Registration Approach for Mobile Imaging Devices Sebastiano Battiato, Senior Member, IEEE, Arcangelo Ranieri Bruna and Giovanni Puglisi Abstract—Digital video stabilization enables to acquire video sequences without disturbing jerkiness by compensating un- wanted camera movements. In this paper we propose a novel fast image registration algorithm based on block matching. Unreliable motion vectors (i.e., not related with jitter movements) are properly filtered out by making use of ad-hoc rules taking into account local similarity, local activity and matching effectiveness. Moreover, a temporal analysis of the relative error computed at each frame has been performed. Reliable information is then used to retrieve inter-frame transformation parameters. Experiments on real cases confirm the effectiveness of the proposed approach even in critical conditions. Index Terms—Video stabilization, motion estimation, block matching. I. I NTRODUCTION I N the last decade, multimedia devices (PDAs, mobile phones, etc.) have been dramatically diffused. Moreover, the increase of their computational performance combined with a higher storage capability allows them to process large amounts of data. These devices, typically small and thin, usually have video acquisition capability. However, making a stable video with these devices is a very challenging task especially when an optical or digital zoom is used. Due to user’s hands shaking, the recorded videos suffer from annoying perturbations. The same problem arises in presence of cameras placed on moving supports (e.g., car, airplane) or fixed cameras operating outdoors. The atmospheric conditions (e.g., the wind) and the vibrations created by passing vehicles make the recorded video unstable. Digital video stabilization enables to acquire video se- quences without disturbing jerkiness by compensating un- wanted camera movements. Video quality is improved and the higher level algorithms present in the device (e.g., segmenta- tion, tracking, recognition) usually work better [1], [2]. More- over, an higher bit rate compression factor can be obtained from the stabilized video than for the unstable one. Many stabi- lization approaches are present in literature and some of them are currently implemented in consumer imaging devices. Some techniques measure camera shake by using mechanical devices (e.g., gyros) and then control the jitter acting on the lens OIS (Optical Image Stabilization) or on the CCD/CMOS sensor. In these approaches both steps (detection and correction) are Copyright (c) 2010 IEEE. Personal use of this material is permitted. However, permission to use this material for any other purposes must be obtained from the IEEE by sending a request to pubs-permissions@ieee.org. Sebastiano Battiato and Giovanni Puglisi are with University of Catania - Italy (e-mail: {battiato,puglisi}@dmi.unict.it). Arcangelo Ranieri Bruna is with STMicroelectronics, AST, Imaging Team, Catania Lab, Italy (e-mail: arcangelo.bruna@st.com). applied before the acquisition, avoiding any post-processing computation and image deformation at the cost of some extra mechanical devices. The computational complexity of the OIS is very low (all the steps are applied before the acquisition) but it requires expensive optical systems and enough space around the objective of the camera, making it difficult to integrate in very small and thin systems like imaging phones. On the other hand, digital video stabilization techniques estimate camera motion making use only of information retrieved from the analysis of the video sequence. They don’t need any additional mechanical apparatus but involve some extra computational cost and introduce the risk of generating image deformation [3]. Furthermore, these approaches may be easily implemented both in real-time and post-processing systems. Digital video stabilization algorithms, in general, are made up of three stages (see [4] for an alternative scheme): global motion estimation, motion filtering and image warping. The first stage is devoted to find the parameters of the transformation occurred between subsequent frames. Translational, similarity and affine are the most commonly adopted motion models. The second step discriminates intentional motion (e.g., panning) from the un- wanted motion (jitter). Typically, motion smoothness is taken into account in the filtering process (jitter is a high frequency signal). The final step consists of the reconstruction of the sta- bilized image through a proper warping. Motion estimation is a crucial video stabilization step. Its accuracy heavily influences motion filtering and image warping. Many conditions, if not properly managed, can degrade its performances: illumination changes, moving objects, image blur, periodic patterns, etc. In this paper we introduce a novel block based image/video registration technique to be used for video stabilization pur- poses. Starting from local motion vectors, through simple and fast rejection rules, our algorithm computes inter-frame trans- formation parameters. Global motion between consecutive frames is estimated considering a two-dimensional similarity model (horizontal and vertical shifts, rotation angle and zoom factor), which is usually a good trade-off between effectiveness and complexity. The rejection criteria are based on local block similarity, local block ”activity” and matching effectiveness. Also a temporal analysis of the involved motion vectors allows to improve the overall robustness of the method. The proposed approach, designed to work in a real-time system, works fine even in critical conditions (illumination changes, moving objects, image blur). The main novelty of the method is related to the choice of pre-filtering criteria, the temporal analysis of block motion vectors coupled with a fairly robust estimator.