Investigation of PET/MRI Image Fusion Schemes for Enhanced Breast Cancer Diagnosis Karl G. Baum, Member, IEEE, Evan Schmidt, Kimberly Rafferty, Maria Helguera, Member, IEEE, David H. Feiglin, and Andrzej Krol, Senior Member, IEEE Abstract–The benefit of registration and fusion of functional images with anatomical images is well appreciated in the advent of PET/CT. There is an increasing interest in expanding this approach to PET/MRI. The focus of much of the related research has been on registering images from different modalities. However, the importance of appropriately jointly displaying (i.e. fusing) the registered images has often been neglected and underestimated. Our aim is to determine which fusion techniques are most useful for enhanced diagnostic performance, ease of use, efficiency, and accuracy for reading registered MRI and PET breast images. Preliminary results indicate that the radiologists were better able to perform a series of tasks when reading the fused PET/MRI data sets using color tables generated by our new genetic algorithm, as compared to commonly used fire/gray or red/gray schemes. I. INTRODUCTION pplication of a multimodality approach is advantageous for detection, diagnosis, and management of many ailments. Obtaining the spatial relationships between the modalities and conveying them to the observer maximizes the benefit that can be achieved. The process of obtaining the spatial relationships and manipulating the images so that corresponding pixels in them represent the same physical location is called image registration. Combining the registered images into a single image is called image fusion. The advantage of a fused image lies in our inability to accurately visually judge spatial relationships between images when they are viewed side by side. Depending on background texture, mottle, shades and colors, identical shapes and lines may appear to be different sizes [1]. This can be demonstrated by well-known simple optical illusions. The most obvious application is to combine a functional image that identifies a region of interest, but lacks structural information necessary for localization, with an anatomical image providing this information. In this paper we examine the benefits of a multimodality approach in the context of breast cancer imaging. We then Manuscript received May 13, 2007. This work was supported in part by the College of Science at the Rochester Institute of Technology, the Chester F. Carlson Center for Imaging Science at the Rochester Institute of Technology, and Kodak. K. G. Baum (email: kgb5056@rit.edu), K. Rafferty, and M. Helguera are with the Chester F. Carlson Center for Imaging Science at the Rochester Institute of Technology, Rochester, NY 14623 USA. E. Schmidt was with the Rochester Institute of Technology, Rochester, NY 14623 USA, during summer break from Honeoye Falls-Lima High School, Honeoye Falls, NY 14472 USA. briefly discuss a recently developed registration technique before launching into possible fusion options. An overview of fusion techniques widely accepted in literature, as well as a novel genetic algorithm-based one are briefly presented. The remainder of the text is devoted to a study in which radiologists were asked to perform a set of tasks reading fused PET/MRI breast images obtained using several different fusion techniques. A. Multimodal Breast Cancer Imaging Application of a multimodality approach is advantageous for detection, diagnosis and management of breast cancer. In this context, F-18-FDG positron emission tomography (PET) [2, 3], and high-resolution and dynamic contrast-enhanced magnetic resonance imaging (MRI) [4, 5] have steadily gained acceptance in addition to x-ray mammography and ultrasonography. Initial experience with combined PET (functional imaging) and x-ray computed tomography (CT, anatomical localization) has demonstrated sizable improvements in diagnostic accuracy, allowing better differentiation between normal (e.g. bowel) and pathological uptake and by providing positive finding in CT images for lesions with low metabolic activity [3]. A method was developed for the coregistration of PET and MRI images, to provide additional information on morphology (including borders, edema, and vascularization) and on dynamic behavior (including fast wash-in, positive enhancement intensity, and fast wash-out) of the suspicious lesion and to allow more accurate lesion localization including mapping of hyper- and hypo-metabolic regions as well as better lesion-boundary definition. Such information might be of great value for grading the breast cancer and assessing the need for biopsy. If biopsy is needed, it could be precisely guided to the most metabolically active (i.e. most malignant) region. II. REGISTRATION Since the breast is entirely composed of soft tissue, it easily deforms and requires nonrigid registration. Physically-based deformable breast models are very difficult to implement because of complex patient-specific breast morphology and highly nonlinear and difficult to measure elastic properties of different types of tissues in the breast, as well as explicitly unknown boundary conditions [6]. The approach presented D. H. Feiglin and A. Krol are with the Radiology Department at SUNY Upstate Medical University, Syracuse, NY 13210 USA. A 2007 IEEE Nuclear Science Symposium Conference Record M19-123 1-4244-0923-3/07/$25.00 ©2007 IEEE. 3774