ORIGINAL PAPER Influence of errors in coordinate transformation due to uncertainties of the design matrix Henry Montecino Castro & Sebastian Santibanez & Sílvio Rogério Correia de Freitas Received: 1 September 2011 / Accepted: 1 July 2013 # Società Italiana di Fotogrammetria e Topografia (SIFET) 2013 Abstract Coordinate transformation between reference sys- tems is a habitual task in geodetic sciences. Several transfor- mation models and adjustment of observations algorithms have been explored. The most popular approach uses the well-known Ordinary Least Squares method (OLS) which seeks to estimate unknown parameters in a linear regression model by minimizing the sum of squared residuals while assuming that the observations are subject to normally dis- tributed random noise. Although this approach has been widely used on many applications, its assumption regarding the distribution of errors is unrealistic. An alternative model created to deal with non normal–random distributions of errors is the Total Least Square model (TLS). Although several approaches have been suggested for representing the errors in the design matrix, most of them fail in including errors with different stochastic characteristics truthfully. This paper presents an assessment of the quality of a Helmert transformation of 2D coordinates when different precisions in both the observation vector and the design matrix exist. Two algorithms were analyzed: the OLS, and the Improved Weighted Total Least Squares (IWTLS). The transformation parameters obtained through IWTLS and OLS show signif- icant differences among them when the variances in the coordinate systems are different. When assessing the trans- formation of coordinates using control points, it becomes clear that both algorithms perform the same in terms of the quality of the transformation. Furthermore, it is shown that an analysis based on the residuals obtained in the least squares adjustment is not a reliable tool to assess the overall quality of the transformation. Keywords Adjustment of observations . Gauss Markov model . Total least squares . Similarity transformation Introduction One of the most habitual tasks in geodetic sciences is the coordinate transformation between different coordinate sys- tems. Nowadays, due to the increasing use of GIS, GPS, and remote sensing tools, the fusion of spatial information tied to different reference system that is affected by uneven precision is becoming an important task (Chen and Hill 2005). A particular example of this situation is the integration of highly detailed cadastral information, generated through terrestrial survey (e.g., total station or GPS) with national standard cartography, generated through airborne photogrammetry. In this context, several models of coordinate transformation 2D and 3D such as rigid body, similarity, orthogonal, polynomial, and affine have been explored (Greenfeld 1997; Bauer and Burkholder 1996; Vanicek and Steeves 1996; Yang 1999; Kutoglu 2004; Yun 2004; Kashani 2006; Awange et al. 2008; Mitsakaki 2004). The least squares model commonly used in parameters estimation is the Gauss–Markov Model (GMM). In this model, the design matrix is assumed as being free of errors, which in real life applications, is rarely true (Felus and Schaffrin 2005). In transformations where both coordinate systems have highly uneven precisions, such as geometric correction of maps, georeferencing of satellite imagery or integration GPS cartography, seems necessary to take into account the precisions related to the observations vector and H. Montecino Castro (*) : S. Santibanez : S. R. C. de Freitas Federal University of Parana, Curitiba, Brazil e-mail: henrymontecino@gmail.com S. Santibanez e-mail: sebastian.santibanez@yahoo.com S. R. C. de Freitas e-mail: sfreitas@ufpr.br Appl Geomat DOI 10.1007/s12518-013-0114-8