Visualization of Fibroid in Laparoscopy Videos using Ultrasound Image Segmentation and Augmented Reality David Kupas University of Debrecen Faculty of Informatics POB 400, 4002 Debrecen, Hungary david_kupass@yahoo.com Andras Hajdu University of Debrecen Faculty of Informatics POB 400, 4002 Debrecen, Hungary hajdu.andras@inf.unideb.hu Peter Torok University of Debrecen Department of Obstetrics and Gynecology POB 37, 4012 Debrecen, Hungary torok.peter@med.unideb.hu Balazs Harangi University of Debrecen Faculty of Informatics POB 400, 4002 Debrecen, Hungary harangi.balazs@inf.unideb.hu Abstract— Though they rarely become malignant, the surgical removal of fibroids (uterine myomas) is commonly considered to prevent any possible future risks. As the least invasive intervention, endoscopic surgery is the most popular approach for this aim. However, since these compact tumors reside in the deep (muscle/connective) tissues of the uterus, they are hardly visible using only the video stream provided by the endoscopic camera. Thus, conform to the current general trend in human surgery, in this paper we propose a multimodal approach to make these tumors more visible during endoscopic interventions, namely, the reconstruction of the whole three-dimensional model of the uterus from ultrasound images and segmentation of the fibroids using this modality. Then, we map the result of the segmentation on the surface of the uterus, hence they become visible during endoscopic surgery. Similar efforts have already been made, considering the usage of MRI for this purpose, but ultrasound image acquisition is more widely available, faster, and cheaper next to the lower image quality. Our aim is to use the output of the 3D ultrasound imaging device during the laparoscopic surgery. Our segmentation pipeline processes the ultrasound images and consists of Otsu’s thresholding using a special mask derived from image averages and morphological snakes to extract uterus boundary. As the final step, we project the segmented 3D model of the uterus with its lesion on an endoscope camera flow in real time to provide an augmented reality application. Keywords— segmentation; uterus; augmented reality; laparoscopy; I. INTRODUCTION Endoscopic surgery of fibroids is a definitive and feasible way of therapy in symptomatic cases. Intramural fibroids located in the deep myometrium cannot be seen with the naked eye, or palpated by laparoscopic or hysteroscopic instruments [1]. In order to avoid unnecessary incisions on the uterine wall, it is important to precisely localize the fibroids pre- and intra- operatively. Augmented reality is used in more and more fields of everyday life [2]. Applying these methods can help in combining preoperative information with endoscopic video. Using MRI has already been published for this purpose [3]. However, the cost of MRI is high and availability is also limited compared to e.g. ultrasonography. As the inclusion of a cheaper and more widely available modality, in this paper we propose a novel technique. Namely, we exhibit how the preoperatively scanned, ultrasonographic data can be projected on the real-time video showing the exact location of the fibroid. As for our methodology during the preoperative work-up a three-dimensional (3D) ultrasound scan is performed of the uterus. The result of this imaging system is stored in DICOM format which contains information about the 3D volume of the uterus. The proposed method can extract and construct automatically the 3D model of the investigated uterus. Next, the fibroid can be marked manually by a surgeon in this model. Finally, using the manually positioned model, the proposed method performs the necessary affine transformations on the 3D uterus model and projects it onto the video frame. In this way, the previously scanned uterus with fibroid in the deep myometrium is projected onto the frames of a video taken during endoscopic surgery to provide an augmented reality application. The workflow of our proposed system can be seen on Fig. 1. Using the advantages of augmented reality, finding and localizing fibroid is easier during endoscopic surgery. Ultrasonography is a widely available technique with lower costs [4]. Preoperative ultrasonographic scanning can help in fibroid identification and localizing procedures. By performing the proper incision, without any other unnecessary ones, complication rate, blood loss and duration of operation can be reduced significantly. The rest of this work is organized as follows. Section II presents the methodology on how we segmented the uterus from DICOM images taken by 3DUS and gives some details about the applied augmented reality. Section III contains our experimental results, in which some conclusions and discussion are drawn as well. II. METHODOLOGY A. Ultrasound Image Extraction The first step of our work is the ultrasound image extraction. We have started with analyzing different types of ultrasound devices. Manufacturers often use their own file format in order to protect their data. These files cannot be opened with external, third-party development kits, only the company can open and work freely with their files. We are limited by these kind of devices. As a result of experimenting with various ultrasound images, we have chosen to process images captured by a Phillips Affiniti