Registration of a 4D Cardiac Motion Model to Endoscopic Video for Augmented Reality Image Guidance of Robotic Coronary Artery Bypass Michael Figl 1,6 , Daniel Rueckert 1 , David Hawkes 3 , Roberto Casula 4 , Mingxing Hu 3 , Ose Pedro 1 , Dong Ping Zhang 1 , Graeme Penney 5 , Fernando Bello 2 , and Philip Edwards 1,2 1 Department of Computing, Imperial College London, UK 2 Department of Biosurgery and Surgical Technology, Imperial College London, UK 3 Centre of Medical Image Computing, University College London, UK 4 Cardiothoracic Surgery, St. Mary’s Hospital, London, UK 5 Division of Imaging Sciences, King’s College London, UK 6 Center for Biomedical Engineering, Medical University of Vienna, Austria Abstract. The aim of the work described in this paper is registration of a 4D preoperative motion model of the heart to the video view of the patient through the intraoperative endoscope, in order to overlay the real video sequence with it. As the heart motion is cyclical it can be modelled using multiple reconstructions of cardiac gated coronary CT. We propose the use of photoconsistency between the two views through the da Vinci endoscope to align to the preoperative heart surface model from CT. We propose averaging of the photoconsistency over the cardiac cycle to improve the registration compared to a single view. Results are presented for simulated renderings and for real video of a beating heart phantom. We found much smoother behaviour of the test function at the minimum when using multiple phases for the registration, furthermore convergence was found to be better when more phases are used. 1 Introduction Totally endoscopic coronary artery bypass (TECAB) is a minimally invasive robotic cardiac procedure performed on the beating heart. The view of the pa- tient is provided by a stereo endoscope which gives the surgeon a 3D view of the operative field. There is a significant proportion of these operations that need to convert to more conventional procedures. This may be due to misidentification of the diseased vessel or difficulty in locating the vessel through fatty deposits on the surface of the myocardium. For this reason there has been some interest in introducing augmented reality (AR) image guidance to TECAB, overlaying these vessels with some preoperative images. Mourgues et al. [6] and Adhami et al [1] proposed such a system where the preoperative model to be overlaid over the real image comes from bilateral x-rays.