Testing the agreement of medical instruments: Overestimation of bias in the Bland–Altman analysis Rafdzah Zaki a, ⁎, Awang Bulgiba a , Noor Azina Ismail b a Julius Centre University of Malaya, Department of Social & Preventive Medicine, Faculty of Medicine, University of Malaya, 50603 Kuala Lumpur, Malaysia b Department of Applied Statistics, Faculty of Economics & Administration, University of Malaya, 50603, Kuala Lumpur, Malaysia abstract article info Available online 11 January 2013 Keywords: Validation studies Statistical data analysis Methodology Bland–Altman method Limits of agreement Objectives. The Bland–Altman method is the most popular method used to assess the agreement of med- ical instruments. The main concern about this method is the presence of proportional bias. The slope of the regression line fitted to the Bland–Altman plot should be tested to exclude proportional bias. The aim of this study was to determine whether the overestimation of bias in the Bland–Altman analysis is still present even when the proportional bias has been excluded. Methods. Data were collected from participants attending a workplace health screening program in a pub- lic university in Malaysia between 2009 and 2010. Variables collected were blood glucose level, body weight and systolic blood pressure (n = 300 per variable). Readings from the original clinical dataset were compared with twenty randomly generated datasets for each variable. The Bland–Altman limits of agreement was used to determine the agreement. The presence of proportional bias was excluded for all datasets using the recommended method. Results. The range of predicted bias was higher than the simulated bias for all datasets. The overestimation of bias increased as the range of actual bias increased. Conclusion. Testing the slope of regression line of the Bland–Altman plot does not remove the artifactual bias in the prediction. © 2013 Elsevier Inc. All rights reserved. Introduction Many important variables measured in medicine are continuous in nature such as blood pressure and oxygen levels. In any clinical situa- tion, either at the stage of health screening, diagnosing cases, or making prognosis, accurate measurement of clinical variables is vital. Numerous new techniques have been developed with the aim of finding a cheaper and safer method to test patients. It is important to be sure that the new method of measurement is in agreement with the current or gold stan- dard method. Agreement signifies the accuracy of that certain instru- ment (de Vet, 1998). The Bland–Altman method is the most popular method used to as- sess the agreement of medical instruments measuring continuous variables (Zaki et al., 2012). Bland and Altman advocated the use of a graphical method (the Bland–Altman plot) and the limits of agree- ment (LoA) (Bland and Altman, 1987, 1995). Despite its popularity, Hopkins (2004) demonstrated that a proportional bias is produced in a Bland–Altman plot. The main concern about the proportional bias is that this will result in overestimation of prediction. The pre- dicted bias will consist of artifactual and real bias, which cannot be differentiated by the researcher (Hopkins, 2004). Thus, the Bland– Altman plot will indicate that bias is present even when there is none. Ludbrook (2010) recommended that a linear regression line be fitted to the Bland–Altman plot (Ludbrook, 2010) to check for this bias. It was argued that, if the slope of the regression line fitted to the Bland–Altman plot is not significantly different from zero then the proportional bias is absent (Ludbrook, 2010). Thus we should not be worried about any artifactual bias. The aim of this study was to determine whether an overestimation of bias in the Bland–Altman analysis will still exist even when the proportional bias is excluded using the linear regression analysis of the Bland–Altman plot. Method Data collection Data were collected from participants attending a workplace health screen- ing program in a large public university. This study was approved by the ethical committee of the university (MEC no: 715.23). A convenient sampling method was applied and written informed consent was obtained from all participants. A total of 300 participants (per variable) consented to be part of this study. Vari- ables collected were blood glucose level, body weight, and systolic blood pres- sure (SBP). Bland (2004) recommended a minimum sample size of 100 for Preventive Medicine 57 (2013) S80–S82 ⁎ Corresponding author. Fax: +60 3 7967 4975. E-mail address: rafdzah@hotmail.com (R. Zaki). 0091-7435/$ – see front matter © 2013 Elsevier Inc. All rights reserved. http://dx.doi.org/10.1016/j.ypmed.2013.01.003 Contents lists available at SciVerse ScienceDirect Preventive Medicine journal homepage: www.elsevier.com/locate/ypmed