IEEE TRANSACTIONS ON MEDICAL IMAGING, VOL. 15, NO. 2, APRIL 1996 zyxwvutsrq 121 Three-Dimensional Multirnodal Image-Guidance for Neurosurgery Terry Peters,* zyxwvutsrq Member, IEEE, Bruce Davey, Member, IEEE, Patrice Munger, Roch Comeau, Alan Evans, and Andr6 Olivier Abstract- We address the use of multimodality imaging as an aid to the planning and guidance of neurosurgical procedures, and discuss the integration of anatomical (CT and MRI), vas- cular (DSA), and functional (PET) data for presentation to the surgeon during surgery. Our workstation is an enhancement of a commercially available system, and in addition to the guidance offered via a hand-held probe, it incorporates the use of mul- timodality imaging and adds enhanced realism to the surgeon through the use of a stereoscopic three-dimensional (3-D) image display. The probe may be visualized stereoscopicallyin single or multimodality images. The integration of multimodality data in this manner provides the surgeon with a complete overview of brain structures on which he is performing surgery, or through which he is passing probes or cannulas, enabling him to avoid critical vessels and/or structures of functional significance. I. INTRODUCTION HE motivation for image guided neurosurgery (IGNS), T in general, is to provide the surgeon with as much information as possible regarding the tissue in the vicinity of the operative site. Some procedures, e.g., stereotactic neu- rosurgery, rely entirely on information gathered from other sources. Even open craniotomy approaches offer the surgeon a limited view of a small area of the cortex. It is imperative therefore that information relating to structure, function, and vascularity also be made accessible to the surgeon. For the last 12 years we have been performing stereotactic neurosurgery at the Montreal Neurological Institute and Hospital (MNI) with the aid of direct image guidance. Initially this involved the use of CT alone [1]-[3], but subsequently DSA and MR images were also employed in the planning process [4]. The use of DSA ensured that the approach to target locations deep within the brain could be made safely without puncturing blood vessels and also served to landmark various cortical structures. We have demonstrated previously that the three-dimensional (3-D) coordinates of targets may be Manuscript received January 10, 1995; revised August 2, 1995. This paper was supported by the Canadian Medical Research Council (MRC), the Canadian National Science and Engineering Research Council (NSERC), and ISG Technologies Inc. The Associate Editor responsible for coordinating the review of this paper and recommending its publication was zyxwvutsrqp S. Pizer. Asterisk indicates corresponding zyxwvutsrqponml author. *T. Peters is with the McConnell Brain Imaging Centre, Montreal Neuro- logical Institute of McGill University, 3801 rue University, MontrBal, QuBbec, H3A 2B4 Canada (e-mail: terry@nil.mni.mcgill.ca). B. Davey is with ISG Technologies Inc. Toronto, Ontario, Canada. P. Munger, R. Comean and A. Evans are with the McConnell Brain Imaging Centre, Montreal Neurological Institute of McGill University, Montrkal, QnCbec H3A 2B4 Canada. A. Olivier is with the Department of Neurosurgery, Montreal Neurological Institute of McGill University, Montrkal, Quebec H3A 2B4 Canada. Publisher Item Identifier S 0278-0062(96)02740-1. determined with acceptable accuracy on the basis of CT and MR images (subject to slice-thickness and geometrical distortion considerations [4], zyxw [5]), and via triangulation based on the target’s perceived position in each of two orthogonal DSA images. [nl order to make these techniques readily available for stereotactic neurosurgery, the image analysis packages devel- oped for each modality were integrated on a personal computer platform such that the target position calculated on the basis of one modality could be easily transferred to the images of the other [4], [5]. 'lie aim of this paper is to demonstrate the effective integra- tion of previously reported stereotactic planning methodology, into an image-guided surgical system incorporating multi- modality visualization and stereoscopic image display of the 3-11 image data sets. We begin with a section describing the relevant background and context within which the work described has taken place. 11. BACKGROUND A. Stereotactic Frames ,4 typical stereotactic apparatus consists of a rigid head- mounted frame, whose purpose is to provide a rigid fiducial marker system during imaging, as well as a rigid platform for mounting instruments during surgery. The set of fiducial markers is visualized with the respective imaging modali- ties and provides a coordinate system within which targets appearing in the images may be localized. The frame also provides the means for determining coordinate transformations between “frame-space’’and “image-space’’as described below. Thle methodology of target localization in tomographic and projection data, based on fiducial markers in the images, is well known and described fully elsewhere [4], [6]. B. 3-0 Visualization It is now routine practice to acquire 3-D data sets from CT and MRI scanners. The display of these data (volumes of vo:tels) is achieved either by projecting surfaces within the volume, or the entire volume itself, onto a viewing screen. These two approaches are known as surface rendering and volume rendering respectively zyxw [7]-[ 101. Today, surface ren- dering is the most common method of displaying 3-D medical images, often in association with texture-mapping to “paint” characteristics of the original data (voxels) onto selected surfaces within the 3-D images. Volume-rendering employs 0278-0062/96$05.00 zyxwvut 0 1996 IEEE