Keith J. Dreyer, DO, PhD Mannudeep K. Kalra, MD, DNB Michael M. Maher, FFR (RCSI), MD, FRCR Autumn M. Hurier, BS Benjamin A. Asfaw, MHSA Thomas Schultz, BS Elkan F. Halpern, PhD James H. Thrall, MD Published online before print 10.1148/radiol.2341040049 Radiology 2005; 234:323–329 Abbreviations: BI-RADS = Breast Imaging Reporting and Data System CI = confidence interval LEXIMER = Lexicon Mediated Entropy Reduction 1 From the Division of Computing and Information Services, Department of Radiology, Massachusetts General Hospital and Harvard Medical School, 100 Charles River Plaza, Suite 471, Cambridge St, Boston, MA 02114. Re- ceived January 9, 2004; revision re- quested March 12; revision received April 6; accepted May 19. Address correspondence to K.J.D. (e-mail: kdreyer@partners.org). Authors stated no financial relation- ship to disclose. Author contributions: Guarantors of integrity of entire study, K.J.D., J.H.T.; study concepts, K.J.D., M.K.K., J.H.T.; study design, K.J.D., M.K.K.; literature research, M.K.K., K.J.D.; clinical studies, K.J.D., M.K.K., M.M.M., B.A.A., T.S.; experimental studies, K.J.D., A.M.H., T.S.; data ac- quisition, M.K.K., M.M.M., A.M.H., B.A.A., T.S., K.J.D.; data analysis/inter- pretation, M.K.K., E.F.H.; statistical analysis, M.K.K., E.F.H.; manuscript preparation, K.J.D., M.K.K., M.M.M., J.H.T., E.F.H.; manuscript definition of intellectual content, K.J.D., M.K.K., J.H.T.; manuscript editing, K.J.D., M.K.K., J.H.T., M.M.M.; manuscript revision/review, K.J.D., M.K.K., M.M.M.; manuscript final version approval, all au- thors © RSNA, 2004 Application of Recently Developed Computer Algorithm for Automatic Classification of Unstructured Radiology Reports: Validation Study 1 PURPOSE: To validate the accuracy of Lexicon Mediated Entropy Reduction (LEXIMER), a new information theory– based computer algorithm developed by the authors for independent analysis and classification of unstructured radiology reports based on the presence of clinically important findings (F T , where T represents “true”) and recommendations for subsequent action (R T ). MATERIALS AND METHODS: The study was approved by the Human Research Committee of the institutional review board. Consecutive de-identified radiology reports (n = 1059) comprising results of barium studies (n = 99), computed tomography (n = 107), mammography (n = 90), magnetic resonance imaging (n = 108), nuclear medicine (n = 99), positron emission tomography (n = 106), radi- ography (n = 212), ultrasonography (n = 131), and vascular procedures (n = 107) were independently analyzed by two radiologists and then with LEXIMER to cate- gorize the reports into F T and F T 0 (containing or not containing clinically important findings) categories and R T and R T 0 (containing or not containing recommendations for subsequent action) categories. Accuracy, sensitivity, specificity, and positive and negative predictive values of LEXIMER for placing reports into F T and F T 0 and R T and R T 0 categories were assessed by using appropriate statistical tests. RESULTS: There was strong interobserver concordance between the two radiolo- gists for placing radiology reports into F T and R T categories (= 0.9, P  .01). For the LEXIMER program, accuracy, sensitivity, specificity, and positive and negative predictive values, respectively, were 97.5% (95% confidence interval [CI]: 96.6%, 98.5%), 98.9% (95% CI: 97.9%, 99.6%), 94.9% (95% CI: 93.1%, 96.0%), 97.5% (95% CI: 96.6%, 98.0%), and 97.7% (95% CI: 95.8%, 98.8%) for placing radiology reports into F T and F T 0 categories and 99.6% (95% CI: 99.2%, 99.9%), 98.2% (95% CI: 95.0%, 99.6%), 99.9% (95% CI: 99.4%, 99.99%), 99.4% (95% CI: 96.3%, 99.9%), and 99.7% (95% CI: 98.9%, 99.9%) for placing reports into R T and R T 0 categories. CONCLUSION: LEXIMER is an accurate automated engine for evaluating the percentage positivity of clinically important findings and rates of recommendation for subsequent action in unstructured radiology reports. © RSNA, 2004 An assessment of diagnostic yield (the frequency of clinically important findings at an imaging examination) and recommendations in radiology practice may help in determin- ing the appropriateness of high-cost and high-volume radiologic procedures. Without the widespread use of structured reporting, most institutions have vast electronic databases of 323 R adiology