Artificial Intelligence in Medicine 62 (2014) 79–90 Contents lists available at ScienceDirect Artificial Intelligence in Medicine j o ur na l ho mepage: www.elsevier.com/locate/aiim From spoken narratives to domain knowledge: Mining linguistic data for medical image understanding Xuan Guo a,∗ , Qi Yu a , Cecilia Ovesdotter Alm b , Cara Calvelli c , Jeff B. Pelz d , Pengcheng Shi a , Anne R. Haake a a College of Computing & Information Sciences, Rochester Institute of Technology, 20 Lomb Memorial Drive, Rochester, NY 14623, USA b College of Liberal Arts, Rochester Institute of Technology, 92 Lomb Memorial Drive, Rochester, NY 14623, USA c College of Health Sciences & Technology, Rochester Institute of Technology, 90 Lomb Memorial Drive, Rochester, NY 14623, USA d Center for Imaging Science, Rochester Institute of Technology, 54 Lomb Memorial Drive, Rochester, NY 14623, USA a r t i c l e i n f o Article history: Received 2 June 2013 Received in revised form 29 July 2014 Accepted 10 August 2014 Keywords: Unified Medical Language System Lexical consensus Semantic relatedness Clustering algorithm Image-based diagnostic reasoning Medical data analysis a b s t r a c t Objectives: Extracting useful visual clues from medical images allowing accurate diagnoses requires physicians’ domain knowledge acquired through years of systematic study and clinical training. This is especially true in the dermatology domain, a medical specialty that requires physicians to have image inspection experience. Automating or at least aiding such efforts requires understanding physicians’ rea- soning processes and their use of domain knowledge. Mining physicians’ references to medical concepts in narratives during image-based diagnosis of a disease is an interesting research topic that can help reveal experts’ reasoning processes. It can also be a useful resource to assist with design of information technologies for image use and for image case-based medical education systems. Methods and materials: We collected data for analyzing physicians’ diagnostic reasoning processes by con- ducting an experiment that recorded their spoken descriptions during inspection of dermatology images. In this paper we focus on the benefit of physicians’ spoken descriptions and provide a general workflow for mining medical domain knowledge based on linguistic data from these narratives. The challenge of a medical image case can influence the accuracy of the diagnosis as well as how physicians pursue the diag- nostic process. Accordingly, we define two lexical metrics for physicians’ narratives—lexical consensus score and top N relatedness score—and evaluate their usefulness by assessing the diagnostic challenge levels of corresponding medical images. We also report on clustering medical images based on anchor concepts obtained from physicians’ medical term usage. These analyses are based on physicians’ spo- ken narratives that have been preprocessed by incorporating the Unified Medical Language System for detecting medical concepts. Results: The image rankings based on lexical consensus score and on top 1 relatedness score are well correlated with those based on challenge levels (Spearman correlation >0.5 and Kendall correlation >0.4). Clustering results are largely improved based on our anchor concept method (accuracy >70% and mutual information >80%). Conclusions: Physicians’ spoken narratives are valuable for the purpose of mining the domain knowledge that physicians use in medical image inspections. We also show that the semantic metrics introduced in the paper can be successfully applied to medical image understanding and allow discussion of additional uses of these metrics. © 2014 Elsevier B.V. All rights reserved. ∗ Corresponding author at: B. Thomas Golisano College of Computing and Infor- mation Sciences, 20 Lomb Memorial Drive, Rochester, NY 14623, USA. Tel.: +1 585 201 2213. E-mail addresses: andyroddick1989@gmail.com, xxg3358@g.rit.edu (X. Guo). 1. Introduction Image understanding is an important topic studied in imaging, computing, and the cognitive sciences and incorporates the domain knowledge of target images, human vision and psychophysics, and data mining. In order to better perform computational image understanding tasks, such as object detection [1–3], shape esti- mation [4], or depth estimation [5], common knowledge from http://dx.doi.org/10.1016/j.artmed.2014.08.001 0933-3657/© 2014 Elsevier B.V. All rights reserved.