Quantitative Evaluation of Image Mosaicing in Multiple Scene Categories Debabrata Ghosh 1 , Sangho Park 1 , Naima Kaabouch 1 , and William Semke 2 1 Department of Electrical Engineering, University of North Dakota, ND, 58203 2 Department of Mechanical Engineering, University of North Dakota, ND, 58203 Abstract- Image mosaicing has been practiced in several computer vision and scientific research areas. There is a clear indication of the advancement of the state of the art of mosaicing algorithms. However, the methods of quantitative evaluation of mosaicing algorithms are still inadequate. Furthermore, a majority of the previous evaluation methodologies lack a sufficient number of performance metrics, while others suffer from computational complication. Therefore, this paper proposes an evaluation method to assess the performance of mosaicing algorithms. This method involves four metrics: percentage of mismatches, difference of pixel intensities, peak signal-to-noise ratio, and mutual information to measure the quality of the mosaicing outputs. These outputs are obtained using a mosaicing algorithm based on the Scale Invariant Feature Transform, Best Bins First, and Random Sample Consensus, reprojection and stitching algorithms. In order to evaluate mosaicing performance objectively, the proposed method compares mosaicing images with the ground-truth images that depict the same scene view. Evaluation has been performed using 36 test sequences from 3 different categories: images of 2D surfaces, images of outdoor 3D scenes, and airborne images from an Unmanned Aerial Vehicle. Exhaustive testing has shown that the proposed metrics are effective in assessing the quality of mosaicing outputs. Keywords: SIFT, Percentage of Mismatches, Difference of Pixel Intensities, Peak Signal-to-Noise Ratio, Mutual Information I. INTRODUCTION Image mosaicing is the stitching of multiple correlated images to generate a larger wide-angle image of a scene. Mosaicing could be regarded as a special case of scene reconstruction where the images are related by planar homography only. Two important situations where consecutive images are exactly correlated by planar homography are: images of a plane produced by a camera which purely rotates about its optical center and images produced by a camera zooming in or out of the scene. These two situations guarantee that images do not show parallax effect, i.e. the scene is approximately planar. Under these circumstances, the underlying scene can be reconstructed from the correlated frames. Several mosaicing algorithms have been proposed over the last decade [1]-[5]. Some novelties include feature-based Image mosaicing proposed by Hu [6], the expectation- maximization algorithm for removing inconsistent overlaid regions in mosaicing proposed by Iiyoshi [7], or the distortion calibration and registration algorithm proposed by Luna [8]. Though the trend of improvement in qualitative performance of stitching algorithms is quite evident, very little work has been done for objective evaluation of this improvement. Applications of image mosaicing in computer vision largely depend on evaluating the quality of the stitching results. In most of the cases the assessment of a mosaicing algorithm is human- based perception. However, as algorithms have become more accurate in recent years, it is not sufficient to rely on visual inspection or subjective evaluation alone, because they might fail to significantly distinguish the mosaics obtained by different algorithms. Scientific evaluation requires mosaic quality to be measured quantitatively, rather than qualitatively. Boutellier et al. [9] created an image sequence from the reference image and then compared the generated mosaic with the reference image. However, they used only one performance metric, which cannot provide sufficient information. Azzari et al. [10] created data sequence from a reference image. They used a software component called Virtual Camera to apply different camera-related effects on the frames to make the testing more realistic. The ground-truth mosaic was generated using the portion of the reference image that was viewed by the Virtual Camera. However, this level of realism is mostly unnecessary because of the computational complexity contributed by this approach. This paper proposes a quantitative evaluation method of any mosaicing algorithm based on several metrics. Performance metrics based on simple pixelwise comparison between the ground truth and the mosaic output are introduced to preserve simplicity in computation.