Constraint Optimisation for Robust Image Matching with Inhomogeneous Photometric Variations and Affine Noise Al Shorin, Georgy Gimel’farb, Patrice Delmas, and Patricia Riddle University of Auckland, Department of Computer Science P.B. 92019, Auckland 1142, New Zealand {al,ggim001,pdel016,pat}@cs.auckland.ac.nz Abstract. While modelling spatially uniform or low-order polynomial contrast and offset changes is mostly a solved problem, there has been limited progress in models which could represent highly inhomogeneous photometric variations. A recent quadratic programming (QP) based matching allows for almost arbitrary photometric deviations. However this QP-based approach is deficient in one substantial respect: it can only assume that images are aligned geometrically as it knows nothing about geometry in general. This paper improves on the QP-based framework by extending it to include a robust rigid registration layer thus increasing both its generality and practical utility. The proposed method shows up to 4 times improvement in the quadratic matching score over a current state-of-the-art benchmark. Keywords: Robust Image Matching, Robust Image Registration, Re- weighted Iterative Least Squares, Affine Functions, Inhomogeneous Pho- tometric Noise, QP, Hildreth-D’Esopo Algorithm 1 Introduction Digital images capture both photometric and geometric properties of a real world 3D scene (from this point forward, for brevity, these properties will be denoted by letters p and g respectively). Matching or registering semantically similar im- ages has to account for their p - and g -dissimilarities or noise (not to be confused with independent random noise which is denoted here as residual noise). These dissimilarities can be caused by a great deal of factors but it is convenient to think of them as being either intrinsic or extrinsic to the scene. The examples of the former type of noise include scene shadows, changing illumination, or dif- ferent object poses, while the latter are most commonly introduced after image acquisition, e.g. brightness or scale adjustments. The third dichotomy of noise introduced here can be used to describe the complexity of dissimilarity patterns between the target and template images: noise patterns can be either homoge- neous (or smooth) or inhomogeneous (or non-smooth). The distinction between those is somewhat arbitrary, but in this paper it is assumed that patterns are