Bayesian approaches to the weighted kappa-like inter-rater agreement measures Journal Title XX(X):2–33 ©The Author(s) 2016 Reprints and permission: sagepub.co.uk/journalsPermissions.nav DOI: 10.1177/ToBeAssigned www.sagepub.com/ SAGE Quoc Duyet Tran 1,2 , Haydar Demirhan 2 and Anil Dolgun 2 Abstract Inter-rater agreement measures are used to estimate the degree of agreement between two or more assessors. When the agreement table is ordinal, different weight functions that incorporate row and column scores are used along with the agreement measures. Selection of row and column scores are effectual on the estimated degree of agreement. The weighted measures are prone to the anomalies frequently seen in agreement tables such has unbalanced table structure or grey zones due to the assessment behaviour of the raters. In this study, Bayesian approaches for the estimation of inter-rater agreement measures are proposed. The Bayesian approaches make it possible to include prior information on the assessment behaviour of the raters in the analysis and impose order-restrictions on the row and column scores. In this way, we improve the accuracy of the agreement measures and mitigate the impact of the anomalies in the estimation of the strength of agreement between the raters. The elicitation of prior distributions is described theoretically and practically for the Bayesian estimation of five agreement measures with three different weights using an agreement table having two grey zones. A Monte Carlo simulation study is conducted to assess the classification accuracy of the Bayesian and classical approaches for the considered agreement measures for a given level of agreement. Recommendations for the selection of the highest performing agreement measure and weight combination are made in the break down of the table structure and sample size. Keywords Agreement table, Prior distribution, Order restriction, Ordinal data, Rater, Weighted kappa. Prepared using sagej.cls [Version: 2017/01/17 v1.20]