A Mixed Reality Guidance System for Robot Assisted Laparoscopic Radical Prostatectomy Abhishek Kolagunda 1(B ) , Scott Sorensen 1 , Sherif Mehralivand 2 , Philip Saponaro 1 , Wayne Treible 1 , Baris Turkbey 2 , Peter Pinto 2 , Peter Choyke 2 , and Chandra Kambhamettu 1 1 University of Delaware, Newark, USA abhi@udel.edu 2 National Institute of Health, Bethesda, USA Abstract. Robotic surgery with preoperative imaging data for planning have become increasingly common for surgical treatment of patients. For surgeons using robotic surgical platforms, maintaining spatial awareness of the anatomical structures in the surgical area is key for good out- comes. We propose a Mixed Reality system which allows surgeons to visualize and interact with aligned anatomical models extracted from preoperative imagery as well as the in vivo imagery from the stereo laparoscope. To develop this system, we have employed techniques to 3D reconstruct stereo laparoscope images, model 3D shape of the anatomical structures from preoperative MRI stack and align the two 3D surfaces. The application we have developed allows surgeons to visualize occluded and obscured organ boundaries as well as other important anatomy that is not visible through the laparoscope alone, facilitating better spatial awareness during surgery. The system was deployed in 9 robot assisted laparoscopic prostatectomy procedures as part of a feasibility study. Keywords: Mixed reality · AR · VR · Robot assisted prostatectomy 1 Introduction Robotic assisted laparoscopic surgery carried out from platforms such as the da Vinci surgical system has been widely used to treat patients. This treatment app- roach uses preoperative imaging for diagnosis and planning purposes. Experts in [19] emphasize that surgery is spatial manipulation, and that a system which can combine information from multiple modalities and present it in way that gives surgeons the best spatial awareness possible would be very useful. Surgeons report using preoperative imaging for surgical planning, to build a mental map of the anatomical structures. They then constantly refer to it during surgery by merging, in their mind, the current laparoscopic view and the preoperative infor- mation to make critical decisions. Our interactions with the surgeons performing c  Springer Nature Switzerland AG 2018 D. Stoyanov et al. (Eds.): OR 2.0/CARE/CLIP/ISIC 2018, LNCS 11041, pp. 164–174, 2018. https://doi.org/10.1007/978-3-030-01201-4_18