(IJACSA) International Journal of Advanced Computer Science and Applications, Vol. 3, No.6, 2012 78 | Page www.ijacsa.thesai.org Resolution Enhancement by Incorporating Segmentation-based Optical Flow Estimation Osama A. Omer Department of Electrical Engineering South Valley University Aswan, Egypt Abstract— In this paper, the problem of recovering a high- resolution frame from a sequence of low-resolution frames is considered. High-resolution reconstruction process highly depends on image registration step. Typical resolution enhancement techniques use global motion estimation technique. However, in general, video frames cannot be related through global motion due to the arbitrary individual pixel movement between frame pairs. To overcome this problem, we propose to employ segmentation-based optical flow estimation technique for motion estimation with a modified model for frame alignment. To do that, we incorporate the segmentation with the optical flow estimation in two-stage optical flow estimation. In the first stage, a reference image is segmented into homogeneous regions. In the second stage, the optical flow is estimated for each region rather than pixels or blocks. Then, the frame alignment is accomplished by optimizing the cost function that consists of L 1 -norm of the difference between the interpolated low-resolution (LR) frames and the simulated LR frames. The experimental results demonstrate that using segmentation-based optical flow estimation in motion estimation step with the modified alignment model works better than other motion models such as affine, and conventional optical flow motion models. Keywords- Optical flow; image segmentation; Horn-Schunck; super resolution; resolution enhancement. I. INTRODUCTION Multi-frame super resolution (SR) is the process of producing a resolution-enhanced frame from multiple low- resolution (LR) frames with sub-pixel shift. SR received much attention in computer vision and image processing communities over the past three decades [1-21] (for review see [6]). SR process typically includes three steps: (i) image registration (motion estimation), (ii) the alignment of LR frames on the high-resolution (HR) grid, and (iii) image restoration. SR methods can be categorized, based on the domain in which the process is done, into time domain [1-5, 7-8, 10-21] and frequency domain [9]. In another way, they can be categorized, based on the incorporated motion model, into global motion model (including translation [3] and affine [2] models) and local motion model (including optical flow [7, 12] and block matching [13]). Furthermore, they can be categorized, based on the alignment process, into non-uniform interpolation [8], deterministic and stochastic regularization [10], and projection onto convex sets (POCS) [11]. Most of the existing super-resolution algorithms [2-3, 9- 11] cannot cope with local motion, because they assume that motion model can be globally parameterized. To overcome the problems of registration error in locally moving parts, three techniques appeared in the literature. The first is to use different global (or local) weights for different registration error levels [14,16]. The second is to use local motion (or multi-motion) estimation to improve the accuracy of registration in the locally moving parts [7, 12, 15, 19]. The third is a combination of the previous two techniques [17, 18]. Among these techniques, the first technique is widely used in SR for its simplicity. The idea of using different weights for different registration is based on rejecting pixels or even whole frames that have high registration error. On the other hand, the main idea behind using local motion estimation techniques is to incorporate information from different frames as much as possible [12, 15, 19]. The main problem of multi- motion estimation [19] is the complexity, since it requires estimation of motion for each moving object in all frames, which is complex and not always accurate because of the fact that the motion of different object affects the motion of other objects. Also, using conventional optical flow estimation for motion estimation [7, 12] is sensitive to noise. In addition, using block-matching results in blocking artefacts due to dividing frames into blocks and incorrectly assuming that all pixels within each block have the same motion vector. Moreover, using combination of local motion estimation and weighting technique adds more complexity to the algorithm as proposed in [17,18]. In addition, algorithm proposed in [15] requires high computational time, since it perform region matching using full search. On the other hand, region segmentation has been employed for image SR in [14, 15, 20, 21]. In [14], segmentation is employed in a region rejection based on the registration inaccuracies. While in [15], region is employed in a region matching in the registration step. In [20], a region-based super-resolution algorithm is proposed in which different filters are used according to the type of region. In this method the segmentation information is not fully used where it is used only to classify regions into homogeneous and inhomogeneous regions. In [21], the image is segmented into background and different objects and each of these are super-resolved separately using a traditional technique and then the super- resolved regions are merged to construct the HR image. This algorithm is very complex since it requires segmentation of moving objects and registration of each object separately.