An Augmented Reality Platform For CABG Surgery Mehrdad Heydarzadeh 1 , Mehrdad Nourani 1 , James Park 2 Abstract—In this paper, we introduce a new platform for in CABG surgery based on marker-less augmented reality technology. Our real-time system captures the surgery scene by a camera, extracts SURF features and matches them in video sequence to track the patient’s heart. It estimates the camera’s pose robustly using RANSAC and highlights the clogged vessel to guide the surgeon during the surgery. I. I NTRODUCTION A. Coronary Artery Bypass Graft Surgery Coronary artery bypass graft (CABG) surgery is a surgical procedure in which one or more blocked coronary arteries are bypassed by a blood vessel graft, usually taken from patient’s arms or legs, to restore normal blood flow to the heart. The surgery relieves chest pain and ischemia, improves the patient’s quality of life, and, in many cases, prolongs the patient’s life [1]. Figure 1 (from left to right) depicts single through quadruple bypass in such a surgery: Coronary arteries usually have an epicardial course, but may have an intramyocardial course. The left anterior de- scending (LAD), considered the gold standard of coronary bypass surgery, takes a deep intramyocardial course [3]. It is difficult to locate the LAD in the case of an intramyocardial LAD because of a thick epicardial adipose tissue or epicar- dial scar tissue. Survey of literature reveals the importance of this problem and a variety of efforts to overcome it [4], [5] and [6]. In [7], authors suggest using epicardial echocardiography with a high-speed linear probe. They apply an echo-pad is applied to the epicardium during the echo-data acquisition. Then, the location of the LAD is determined with reference to stabalizer arms and depth from epicardium. In [8], the LAD is located by sticking a radiopaque marker to its surface using anterade cardioplegia cannula and taking cineangio- graphic images is offered for locating the LAD. In [9], the authors propose a hand-held epicardial ultra-sonic doppler flow detector for finding the target vessel. The authors in [10] compare these methods and discuss their disadvantages. Most of the suggested methods are based on existing devices for detecting flow rate in vessels. To the best of our knowledge, oo standalone technology has been proposed so far. B. Augmented Reality for Computer Aided Surgery Augmented reality (AR) technology overlays digital in- formation on a real-time image. The authors of [11] require *This work is partially supported by TxMRC consortium 1 Department of Electrical Engineering and Computer Science, University of Texas at Dallas, {mehhey,nourani}@utdallas.edu 2 Texas Health Presbyterian Hospital Dallas, jamespark@ texashealth.org Fig. 1. Single to quadruple CABG (from [2]) three characteristics for an AR system: (1) combine real and virtual (2) interactive in real time (3) registered in 3D. AR applications can be divided into two groups: marker- based (MB-AR) and marker-less (ML-AR). MB-AR uses a reference object (fiducial) for tracking an object of interest and estimating the relative camera’s position (pose). ML- AR doesn’t use any fiducial in a scene and it estimate camera’s pose using existing objects in the scene. So, ML- AR approach is more desirable. However, tracking and pose estimation are more challenging in ML-AR. One of the applications of AR technology is computer aided surgery (CAS). For example, the authors of [12] suggest using AR is in laparoscopic surgery. Authors in [13] used AR for image guided breast surgery. C. Key Contribution The exact location of the clogged vessel is detected during pre-surgery x-ray or magnetic field resonance imaging (MRI). A question arises: “If we know the position of the clog relative to some visible and easily detectable points, can we estimate the position of clogged vessel?” We answer this question by developing an ML-AR system. Our system uses a 3D model of a patient’s heart to estimate the position of the clog relative to visible parts. This 3D model can be obtained using pre-operation 3D biomedical imaging technologies like 3D x-ray and MRI. Our platform finds the visible parts of the heart in the surgery scene, registers the heart according to the 3D model and produces overlay images for highlighting the clog’s position in real-time. II. SYSTEM ARCHITECTURE The proposed system has six modules, shown in Figure 2 which are introduced next. A. Camera We use a pinhole camera in our platform. Although pin- hole cameras are inexpensive and widely available, they add 978-1-4799-7234-0/15/$31.00 ©2015 IEEE