A modality conversion approach to MV-DRs and KV-DRRs registration using information bottlenecked conditional generative adversarial network Cong Liu Ningbo Institute of Technology, Zhejiang University, Ningbo 315100, China Zheming Lu School of Aeronautics and Astronautics, Zhejiang University, Hangzhou 310058, China Longhua Ma, and Lang Wang Ningbo Institute of Technology, Zhejiang University, Ningbo 315100, China Xiance Jin Radiation and Medical Oncology Department, 1st Affiliated Hospital of Wenzhou Medical University, Wenzhou 325003, China Wen Si a) Internet of Things College, Shanghai Business School, Shanghai 201400, China Huashan Hospital, Fudan University, Shanghai 200031, China (Received 10 May 2019; revised 7 August 2019; accepted for publication 8 August 2019; published 6 September 2019) Purpose: As affordable equipment, electronic portal imaging devices (EPIDs) are wildly used in radiation therapy departments to verify patients’ positions for accurate radiotherapy. However, these devices tend to produce visually ambiguous and low-contrast planar digital radiographs under mega- voltage x ray (MV-DRs), which poses a tremendous challenge for clinicians to perform multimodal registration between the MV-DRs and the kilovoltage digital reconstructed radiographs (KV-DRRs) developed from the planning computed tomography. Furthermore, the existent of strong appearance variations also makes accurate registration beyond the reach of current automatic algorithms. Methods: We propose a novel modality conversion approach to this task that first synthesizes KV images from MV-DRs, and then registers the synthesized and real KV-DRRs. We focus on the synthesis technique and develop a conditional generative adversarial network with information bottleneck extension (IB-cGAN) that takes MV-DRs and nonaligned KV-DRRs as inputs and outputs synthesized KV images. IB-cGAN is designed to address two main challenges in deep- learning-based synthesis: (a) training with a roughly aligned dataset suffering from noisy correspon- dences; (b) making synthesized images have real clinical meanings that faithfully reflects MV-DRs rather than nonaligned KV-DRRs. Accordingly, IB-cGAN employs (a) an adversarial loss to provide training supervision at semantic level rather than the imprecise pixel level; (b) an IB to constrain the information from the nonaligned KV-DRRs. Results: We collected 2698 patient scans to train the model and 208 scans to test its performance. The qualitative results demonstrate realistic KV images can be synthesized allowing clinicians to per- form the visual registration. The quantitative results show it significantly outperforms current non- modality conversion methods by 22.37% (P = 0.0401) in terms of registration accuracy. Conclusions: The modality conversion approach facilitates the downstream MV–KV registration for both clinicians and off-the-shelf registration algorithms. With this approach, it is possible to benefit the developing countries where inexpensive EPIDs are widely used for the image-guided radiation therapy. © 2019 American Association of Physicists in Medicine [https://doi.org/10.1002/mp.13770] Key words: generative adversarial networks, image-guided radiation therapy, image synthesis, information bottleneck, multimodal image registration 1. INTRODUCTION The accurate patient positioning before the irradiation is an essential procedure for delivering a tumoricidal radiation dose while minimizing dose to the surrounding normal tis- sues. Modern image-guided radiation therapy (IGRT) verifies patient positions by comparing the in-room images taken before the treatment to the reference computed tomography (CT) scans taken during the planning phase. Such a process is also called image registration where two images are aligned with each other. There are several in-room imaging technolo- gies which can be used, such as in-room x rays, cone beam CT (CBCT), and electronic portal imaging devices (EPIDs) using either kilovoltage (KV) or megavoltage (MV) x rays. Among them, MV-EPIDs are wildly used because they take the treatment beam as the imaging source and are offered as standard equipment by nearly all linear accelerator (LINAC) vendors. The images acquired by MV-EPID are orthogonal 4575 Med. Phys. 46 (10), October 2019 0094-2405/2019/46(10)/4575/13 © 2019 American Association of Physicists in Medicine 4575