Article Validating Translation Test Items via the Many-Facet Rasch Model Wen-Ta Tseng Department of Applied Foreign Languages, National Taiwan University of Science and Technology, Taipei City, Taiwan Tzi-Ying Su National Taiwan Normal University, Taipei City, Taiwan John-Michael L. Nix Center for General Education, National Taitung University, Taitung County, Taiwan Abstract This study applied the many-facet Rasch model to assess learners’ translation ability in an English as a foreign language context. Few attempts have been made in extant research to detect and calibrate rater severity in the domain of translation testing. To fill the research gap, this study documented the process of validating a test of Chinese-to-English sentence translation and modeled raters’ scoring propensity defined by harshness or leniency, expert/novice effects on severity, and concomitant effects on item difficulty. Two hundred twenty-five, third-year senior high school Taiwanese students and six educators from tertiary and secondary educational insti- tutions served as participants. The students’ mean age was 17.80 years (SD ¼ 1.20, range 17–19). The exam consisted of 10 translation items adapted from two entrance exam tests. The results showed that this subjectively scored performance assessment exhibited robust unidimensionality, thus reliably measuring translation ability free from unmodeled disturbances. Furthermore, discrepancies in ratings between novice and expert raters were also identified and modeled by the many- facet Rasch model. The implications for applying the many-facet Rasch model in translation tests at the tertiary level were discussed. Keywords Translation test, rater bias, many-facet Rasch model Psychological Reports 0(0) 1–25 ! The Author(s) 2018 Reprints and permissions: sagepub.com/journalsPermissions.nav DOI: 10.1177/0033294118768664 journals.sagepub.com/home/prx Corresponding Author: Wen-Ta Tseng, Department of Applied Foreign Languages, National Taiwan University of Science and Technology, Taipei City, Taiwan. Email: wenta.tseng@mail.ntust.edu.tw