Robust Learning-Based Annotation of Medical Radiographs Yimo Tao 1,2 , Zhigang Peng 1 , Bing Jian 1 , Jianhua Xuan 2 , Arun Krishnan 1 , and Xiang Sean Zhou 1 1 CAD R&D, Siemens Healthcare, Malvern, PA USA 2 Dept. of Electrical and Computer Engineering, Virginia Tech, Arlington, VA USA Abstract. In this paper, we propose a learning-based algorithm for au- tomatic medical image annotation based on sparse aggregation of learned local appearance cues, achieving high accuracy and robustness against se- vere diseases, imaging artifacts, occlusion, or missing data. The algorithm starts with a number of landmark detectors to collect local appearance cues throughout the image, which are subsequently verified by a group of learned sparse spatial configuration models. In most cases, a decision could already be made at this stage by simply aggregating the verified detections. For the remaining cases, an additional global appearance fil- tering step is employed to provide complementary information to make the final decision. This approach is evaluated on a large-scale chest ra- diograph view identification task, demonstrating an almost perfect per- formance of 99.98% for a posteroanterior/anteroposterior (PA-AP) and lateral view position identification task, compared with the recently re- ported large-scale result of only 98.2% [1]. Our approach also achieved the best accuracies for a three-class and a multi-class radiograph annotation task, when compared with other state of the art algorithms. Our algo- rithm has been integrated into an advanced image visualization worksta- tion, enabling content-sensitive hanging-protocols and auto-invocation of a computer aided detection algorithm for PA-AP chest images. 1 Introduction The amount of medical image data produced nowadays is constantly growing, and a fully automatic image content annotation algorithm can significantly im- prove the image reading workflow, by automatic configuration/optimization of image display protocols, and by off-line invocation of image processing (e.g., de- noising or organ segmentation) or computer aided detection (CAD) algorithms. However, such annotation algorithm must perform its tasks in a very accurate and robust manner, because even “occasional” mistakes can shatter users’ con- fidence in the system, thus reducing its usability in the clinical settings. In the radiographic exam routine, chest radiograph comprise at least one-third of all di- agnostic radiographic procedures. Chest radiograph provides sufficient patholog- ical information about cardiac size, pneumonia-shadow, and mass-lesions, with low cost and high reproducibility. However, about 30%-40% of the projection B. Caputo et al. (Eds.): MCBR CDS 2009, LNCS 5853, pp. 77–88, 2010. Springer-Verlag Berlin Heidelberg 2010