VR-Caps: A Virtual Environment for Capsule Endoscopy Kağan İncetan a , Ibrahim Omer Celik b,2 , Abdulhamid Obeid a,2 , Guliz Irem Gokceler a , Kutsev Bengisu Ozyoruk a , Yasin Almalioglu c , Richard J. Chen d,g , Faisal Mahmood d,e,i , Hunter Gilbert h , Nicholas J. Durr i , Mehmet Turan a,* a Institute of Biomedical Engineering, Bogazici University, Istanbul, Turkey b Department of Computer Engineering, Bogazici University, Istanbul, Turkey c Computer Science Department, University of Oxford, Oxford, UK d Brigham and Women’s Hospital, Harvard Medical School, Boston, MA, USA e Cancer Data Science, Dana Farber Cancer Institute, Boston, MA, USA f Cancer Program, Broad Institute of Harvard and MIT, Cambridge, MA, USA g Department of Biomedical Informatics, Harvard Medical School, Boston, MA, USA h Deparment of Mechanical and Industrial Engineering, Louisiana State University, Baton Rouge, LA USA i Department of Biomedical Engineering, Johns Hopkins University (JHU), Baltimore, MD, USA Abstract Current capsule endoscopes and next-generation robotic capsules for diagnosis and treatment of gastrointestinal diseases are complex cyber-physical platforms that must orchestrate complex software and hardware functions. The desired tasks for these systems include visual localization, depth estimation, 3D mapping, disease detection and segmentation, automated navigation, active control, path realization and optional therapeutic modules such as targeted drug delivery and biopsy sampling. Data-driven algorithms promise to enable many advanced functionalities for capsule endoscopes, but real-world data is challenging to obtain. Physically-realistic simulations providing synthetic data have emerged as a solution to the development of data-driven algorithms. In this work, we present a comprehensive simulation platform for capsule endoscopy operations and introduce VR-Caps, a virtual active capsule environment that simulates a range of normal and abnormal tissue conditions (e.g., inflated, dry, wet etc.) and varied organ types, capsule endoscope designs (e.g., mono, stereo, dual and 360°camera), and the type, number, strength, and placement of internal and external magnetic sources that enable active locomotion. VR-Caps makes it possible to both independently or jointly develop, optimize, and test medical imaging and analysis software for the current and next-generation endoscopic capsule systems. To validate this approach, we train state-of-the-art deep neural networks to accomplish various medical image analysis tasks using simulated data from VR-Caps and evaluate the performance of these models on real medical data. Results demonstrate the usefulness and effectiveness of the proposed virtual platform in developing algorithms that quantify fractional coverage, camera trajectory, 3D map reconstruction, and disease classification. All of the code, pre-trained weights and created 3D organ models of the virtual environment with detailed instructions how to setup and use the environment are made publicly available at https://github.com/CapsuleEndoscope/VirtualCapsuleEndoscopy and a video demonstration can be seen at https://www.youtube.com/watch?v=UQ2u3CIUciA. Keywords: Capsule Endoscopy, Deep Reinforcement Learning, Area Coverage, Disease Classification, Synthetic Data Generation 1. Introduction Optical colonoscopy is considered to be the gold stan- dard in the early prognosis, diagnosis and intervention of critical upper and lower GI-tract diseases such as colorectal cancer (CRC), Crohn’s disease, ulcerative colitis, hemor- rhoids or inflammation. Despite colonoscopies demonstrat- ing clinical impact in reducing CRC incidence, the current standard of care for patient screening is invasive and has ∗ Corresponding Author Email address: mehmet.turan@boun.edu.tr (Mehmet Turan) 1 This work was supported by the Scientific and Technological Research Council of Turkey (TUBITAK) with grant 2232 - The International Fellowship for Outstanding Researchers 2 These authors contributed equally poor sensitivity in detecting the adenomatous polyps. With an estimated 19 million colonoscopies performed annually in the United States and 6-28% polyps missed in routine screenings, CRC is the second most prevalent cancer and leading cause of cancer death (Lee et al., 2017). Due to the fact that small intestines are difficult to access with con- ventional endoscopes and since patients suffer from heavy pain and discomfort during traditional endoscopy, technolo- gies such as Wireless Capsule Endoscopes (WCEs) have emerged for navigating the entire gastrointestinal tract and identifying adenomatous polyps and other non-polyploid lesions, which would preclude the progression of CRC. Unlike conventional endoscopes, WCEs are swallowable, pill-like imaging devices that allow for direct visualizations of the GI-tract without requiring bowel preparation or Preprint submitted to Medical Image Analysis September 1, 2020 arXiv:2008.12949v1 [cs.CV] 29 Aug 2020