J. P. Marcin M. M. Pollack K. M. Patel U. E. Ruttimann Decision support issues using a physiology based score Received: 15 May 1998 Accepted: 9 September 1998 PRISM III and updated PRISM algorithms are copyrighted and may be the subject of one or more patents held by Children's Research Institute. The equations are available without charge for research uses including the independent verification of their accuracy and reliability. Children's National Medical Center may receive com- pensation resulting from non-research uses of PRISM III and PRISM algorithms. Supported, in part, by grant MCH-110584 from the Maternal and Child Health Bu- reau (Title V, Social Security Act), Health Resources and Services Administration, Department of Health and Human Servi- ces and by an independent effort of mem- bers of the National Association of Chil- dren's Hospitals and Related Institutions J. P.Marcin × M. M. Pollack ( ) ) George Washington University School of Medicine, Department of Critical Care Medicine, Children's National Medical Center, 111 Michigan Avenue, NW, Washington, DC 20010-2970, USA email: mpollack@cnmc.org Tel.: +1 (202) 8 84-21 30 Fax: +1 (2 02) 8 84-57 24 K. M. Patel George Washington University School of Medicine, Center for Health Service and Clinical Research, Children's Research Institute, Children's National Medical Center, Washington, DC, USA U. E. Ruttimann George Washington University School of Medicine, National Institute on Alcohol Abuse and Alcoholism, National Institutes of Health, Bethesda, MD, USA Abstract Objective: As physiology based assessments of mortality risk become more accurate, their poten- tial utility in clinical decision sup- port and resource rationing deci- sions increases. Before these predic- tion models can be used, however, their performance must be statisti- cally evaluated and interpreted in a clinical context. We examine the is- sues of confidence intervals (as esti- mates of survival ranges) and confi- dence levels (as estimates of clinical certainty) by applying Pediatric Risk of Mortality III (PRISM III) in two scenarios: (1) survival predic- tion for individual patients and (2) resource rationing. Design: A non-concurrent cohort study. Setting: 32 pediatric intensive care units (PICUs). Patients: 10 608 consecutive patients (571 deaths). Interventions: None. Measurements and results: For the individual patient application, we investigated the observed survival rates for patients with low survival predictions and the confidence in- tervals associated with these predic- tions. For the resource rationing ap- plication, we investigated the maxi- mum error rate of a policy which would limit therapy for patients with scores exceeding a very high thresh- old. For both applications, we also investigated how the confidence in- tervals change as the confidence le- vels change. The observed survival in the PRISM III groups > 28, > 35, and > 42 were 6.3, 5.3, and 0 %, with 95 % upper confidence interval bounds of 10.5, 13.0, and 13.3 %, re- spectively. Changing the confidence level altered the survival range by more than 300 % in the highest risk group, indicating the importance of clinical certainty provisions in prog- nostic estimates. The maximum er- ror rates for resource allocation de- cisions were low (e. g., 29 per 100 000 at a 95 % certainty level), equivalent to many of the risks of daily living. Changes in confidence level had rel- atively little effect on this result. Conclusions: Predictions for an in- dividual patient's risk of death with a high PRISM score are statistically not precise by virtue of the small number of patients in these groups and the resulting wide confidence intervals. Clinical certainty (confi- dence level) issues substantially in- fluence outcome ranges for individ- ual patients, directly affecting the utility of scores for individual pa- tient use. However, sample sizes are sufficient for rationing decisions for many groups with higher certainty levels. Before there can be wide- spread acceptance of this type of decision support, physicians and families must confront what they believe is adequate certainty. Key words Severity of illness × Pediatric intensive care × Intensive care units × PRISM × Prediction × Certainty Intensive Care Med (1998) 24: 1299±1304 Ó Springer-Verlag 1998 ORIGINAL