Assessing and quantifying inter-rater variation for dichotomous ratings using a Rasch model Jørgen Holm Petersen 1, * , Klaus Larsen 2 , and Svend Kreiner 1 1 Department of Biostatistics, University of Copenhagen, Øster Farimagsgade 5 entrance B, Postboks 2099, D-1014 København, Denmark 2 Clinical Research Unit, Hvidovre Hospital, University of Copenhagen, Kettegaards All´ e 30, DK-2650 Hvidovre, Denmark SUMMARY We present a new model-based approach to the analysis of agreement between raters in a situation where all raters have supplied dichotomous ratings of the same cases in a sample. The model is logistic regression model with random effects - a Rasch model. In the rater setting, the Rasch model includes parameters that allow raters to have different propensity to score a given set of individuals positively or negatively - the rater bias. An exact score test of the hypothesis of no rater bias is proposed and is shown to be an exact generalized McNemar’s test. Based on the model, we suggest quantifying the rater variation as a suitable measure of the variation of the rater odds ratios. An important example that will serve to motivate and illustrate the proposed model, is the study of Umbilical artery Doppler velocimetry used by obstetricians to assess the status of a fetus. The purpose of the assessment is to improve the fetus’ chance of survival by choosing the optimal time of elective delivery. In the study, * Correspondence to: Jørgen Holm Petersen, Department of Biostatistics, University of Copenhagen, Øster Farimagsgade 5 entrance B, Postboks 2099, D-1014 København, Denmark, Email: J.H.Petersen@biostat.ku.dk