Authors: Andrew Worster, MD, MSc, Arlene Sardo, RN, ACNP, Kevin Eva, PhD, Christopher M.B. Fernandes, MD, and Suneel Upadhye, MD, MSc, Hamilton, Ontario, Canada Andrew Worster is Research Director, Division of Emergency Medicine, McMaster University and Hamilton Health Sciences, Department of Emergency Medicine, Hamilton, Ontario, Canada. Arlene Sardo is Nurse Practitioner, Hamilton Health Sciences, Department of Emergency Medicine, Hamilton, Ontario, Canada. Kevin Eva is Epidemiologist, Department of Clinical Epidemiology and Biostatics, McMaster University, Hamilton, Ontario, Canada. Christopher M.B. Fernandes is Professor of Emergency Medicine, University of Western Ontario, London, Ontario. Suneel Upadhye is Undergraduate Coordinator, Division of Emergency Medicine, McMaster University and Hamilton Health Sciences, Department of Emergency Medicine, Hamilton, Ontario, Canada. This study was presented at the 2005 Society of Academic Emergency Medicine Annual Meeting, May 22-25, 2005, New York, NY; the Canadian Association of Emergency Physicians Annual Scientific Meeting, May 30, 2005, Edmonton, Alberta; and the ENA Annual Conference, Nashville, Tenn, September 14-17, 2005. Funded by a research grant to Dr. Andrew Worster from the Hamilton Emergency Services Network, Hamilton Ontario, Canada. For correspondence, write: Dr. Christopher M.B. Fernandes, London Health Sciences, London, Ontario, Canada N6A 5W9; E-mail: christopher_fernandes@sympatico.ca. J Emerg Nurs 2007;33:319-23. 0099-1767/$32.00 Copyright n 2007 by the Emergency Nurses Association. doi: 10.1016/j.jen.2006.12.016 Background: Published studies of triage scale inter-rater re- liability assessment have been conducted mostly using paper case scenarios. Objective: To determine if this method of inter-rater reliability assessment generated significantly different measures from those generated from live triage cases. Methods: This was a multicenter, prospective, observational cohort study of a population-based random sample of patients triaged at 2 emergency departments during a period of 4 months. All patients presenting to the emergency department within the study periods were simultaneously and indepen- dently triaged using a 5-level triage acuity scale by 2 to 3 research triage nurses blinded to each other’s assessment and to the study objective. After 6 months, the same nurses were asked to assign triage scores to paper case scenarios of the same patients that they had each previously triaged. Results: Each of the 9 research nurses triaged approximately 90 cases. The inter-rater reliabilities as measured by an intraclass correlation coefficient were 0.9 (95% CI = 0.87 to 93) for the live triage assessments and 0.76 (95% CI = 0.73 to 0.79) for the paper case scenarios. The mean triage score assigned to the live cases was significantly less than that assigned to the paper-based cases (3.17; 95% CI = 3.08 to 3.26) (p b 0.001). Conclusions: There is moderate to high agreement between live cases and paper case scenarios, and the inter-rater reliabilities, although significantly different, are acceptable in both cases. It is impossible to determine which triage setting provides a more accurate triage score but paper case scenarios generally receive lower triage scores than live cases. Triage Tool Inter-rater Reliability: A Comparison of Live Versus Paper Case Scenarios RESEARCH August 2007 33:4 JOURNAL OF EMERGENCY NURSING 319