COMMENT The Relationship Between Mean Square Differences and Standard Error of Measurement: Comment on Barchard (2012) Tianshu Pan NCS Pearson, Inc., San Antonio, Texas Yue Yin University of Illinois at Chicago In the discussion of mean square difference (MSD) and standard error of measurement (SEM), Barchard (2012) concluded that the MSD between 2 sets of test scores is greater than 2(SEM) 2 and SEM underestimates the score difference between 2 tests when the 2 tests are not parallel. This conclusion has limitations for 2 reasons. First, strictly speaking, MSD should not be compared to SEM because they measure different things, have different assumptions, and capture different sources of errors. Second, the related proof and conclusions in Barchard hold only under the assumptions of equal reliabilities, homogeneous variances, and independent measurement errors. To address the limitations, we propose that MSD should be compared to the standard error of measurement of difference scores (SEM X-Y ) so that the comparison can be extended to the conditions when 2 tests have unequal reliabilities and score variances. Keywords: mean square difference, reliability, standard error of measurement In a recent article, Barchard (2012) criticized that the existing testing theory cannot directly provide information about how much scores would change if slightly different measurement conditions were used. To address this void, Barchard introduced two statistics for evaluating score consistency, that is, the root mean square difference (RMSD) and the concordance correlation coefficient (CCC; Lin, 1989), and discussed the relationship of the RMSD and CCC to the intraclass correlation coefficients, product–moment correlation, and standard error of measurement (SEM). Barchard has made an important contribution to testing theory by highlight- ing the needs of statistics for capturing changes in score, introduc- ing possible statistics, and addressing their relationship to com- monly used indices in classical test theory. However, Barchard’s outline and discussion of the relationship between MSD and SEM suffer from limitations. In the article by Barchard (2012), RMSD(A, 1) denotes an absolute definition of agreement and is estimated on the basis of raw data, and RMSD(C, 1) denotes a consistency definition of agreement and is estimated on the basis of mean-deviated data. Mean square difference (MSD) is RMSD squared. Given two sets of test scores X and Y, MSD(A, 1) and MSD(C, 1) are defined as follows: MSDA, 1 =  X - Y 2 N =  X 2 +  Y 2 - 2 XY +  X -  Y  2 (1) and MSDC, 1 =  X-Y 2 =  X 2 +  Y 2 - 2 XY =  X 2 +  Y 2 - 2 XY  X  Y , (2) where N is the sample size,  X and  Y are the standard deviations of X and Y, respectively,  X and  Y are the means of X and Y, respectively,  XY is the covariance between X and Y,  XY is the correlation coefficient between X and Y, and  X-Y is the standard deviation of difference scores between X and Y. In classical test theory, SEM is estimated as SEM =   1 - , (3) where  is the standard deviation of test scores and  is the reliability coefficient. Based on Equations 1 to 3, Barchard (2012) obtained the fol- lowing relationships when the two tests are not parallel, MSDA, 1  MSDC, 1  2SEM 2 , (4) and when parallelism is assumed, MSDA, 1 = MSDC, 1 = 2SEM 2 . (5) Barchard (2012, p. 308) further concluded that the traditional SEM “gives an overly optimistic impression of the precision of measurement.” However, the derivation in the connection between MSD and SEM, and conclusions by Barchard suffer from two general limitations—incompatibility of MSD and SEM, and the calculation of SEM—with each general limitation containing sub- parts. Incompatibility of MSD and SEM Strictly speaking, MSD and SEM should not be compared directly because they require different assumptions, measure dif- ferent things, and capture different sources of variations. Tianshu Pan, NCS Pearson, Inc., San Antonio, Texas; Yue Yin, Depart- ment of Educational Psychology, College of Education, University of Illinois at Chicago. Correspondence concerning this article should be addressed to Tianshu Pan, NCS Pearson, Inc., 19500 Bulverde Road, San Antonio, TX 78259- 3701. E-mail: Tianshu.Pan@Pearson.com Psychological Methods © 2012 American Psychological Association 2012, Vol. 17, No. 2, 309 –311 1082-989X/12/$12.00 DOI: 10.1037/a0028250 309 This document is copyrighted by the American Psychological Association or one of its allied publishers. This article is intended solely for the personal use of the individual user and is not to be disseminated broadly.