RadSem: Semantic Annotation and Retrieval for Medical Images Manuel M¨ oller 1 , Sven Regel 2 , and Michael Sintek 1 1 German Research Center for Artificial Intelligence (DFKI) GmbH Kaiserslautern, Germany manuel.moeller@dfki.de , michael.sintek@dfki.de 2 Chemnitz University of Technology, Chemnitz, Germany sven.regel@cs.tu-chemnitz.de Abstract. We present a tool for semantic medical image annotation and retrieval. It leverages the MEDICO ontology which covers formal background information from various biomedical ontologies such as the Foundational Model of Anatomy (FMA), terminologies like ICD-10 and RadLex and covers various aspects of clinical procedures. This ontology is used during several steps of annotation and retrieval: (1) We developed an ontology-driven metadata extractor for the medical image format DI- COM. Its output contains, e. g., person name, age, image acquisition parameters, body region, etc. (2) The output from (1) is used to sim- plify the manual annotation by providing intuitive visualizations and to provide a preselected subset of annotation concepts. Furthermore, the extracted metadata is linked together with anatomical annotations and clinical findings to generate a unified view of a patient’s medical history. (3) On the search side we perform query expansion based on the structure of the medical ontologies. (4) Our ontology for clinical data management allows us to link and combine patients, medical images and annotations together in a comprehensive result list. (5) The medical annotations are further extended by links to external sources like Wikipedia to provide additional information. 1 1 Introduction Advances in medical imaging have enormously increased the volume of digital images produced in clinical practice. At the same time, modern hospital infor- mation systems have also become more complex. Radiological findings are kept separately from images which, in turn, are kept separately from patient account- ing and billing information. Currently, these systems are more or less isolated from each other and do not allow queries to span across these systems. Thus it has become challenging for clinicians to query for and retrieve relevant historical data due to the volume 1 This research has been supported in part by the THESEUS Program in the MEDICO Project, which is funded by the German Federal Ministry of Economics and Tech- nology under the grant number 01MQ07016. The responsibility for this publication lies with the authors. L. Aroyo et al. (Eds.): ESWC 2009, LNCS 5554, pp. 21–35, 2009. c  Springer-Verlag Berlin Heidelberg 2009