Phase Collaborative Network for Two-Phase Medical Imaging Segmentation Huangjie Zheng 1,3* , Lingxi Xie 2,4 , Tianwei Ni 3 , Ya Zhang 1 , Yan-Feng Wang 1 , Qi Tian 4 Elliot K. Fishman 5 , Alan L. Yuille 2 1 Shanghai Jiao Tong University 2 The Johns Hopkins University 3 University of Texas at Austin 4 Carnegie Mellon University 5 Huawei Noahs Ark Lab 6 The Johns Hopkins School of Medicine {zhj865265, ya zhang, wangyanfeng}@sjtu.edu.cn {198808xc, twni2016, alan.l.yuille}@gmail.com huangjie.zheng@utexas.edu tian.qi1@huawei.com efishman@jhmi.edu Abstract In real-world practice, medical images acquired in dif- ferent phases possess complementary information, e.g., ra- diologists often refer to both arterial and venous scans in order to make the diagnosis. However, in medical image analysis, fusing prediction from two phases is often diffi- cult, because (i) there is a domain gap between two phases, and (ii) the semantic labels are not pixel-wise corresponded even for images scanned from the same patient. This paper studies organ segmentation in two-phase CT scans. We pro- pose Phase Collaborative Network (PCN), an end-to-end framework that contains both generative and discrimina- tive modules. PCN can be mathematically explained to for- mulate phase-to-phase and data-to-label relations jointly. Experiments are performed on a two-phase CT dataset, on which PCN outperforms the baselines working with one- phase data by a large margin, and we empirically verify that the gain comes from inter-phase collaboration. Besides, PCN transfers well to two public single-phase datasets, demonstrating its potential applications. 1. Introduction Semantic segmentation is one of the fundamental prob- lems in computer vision which implies a wide range of ap- plications. Recent years, with the rapid development of deep learning [28, 27, 44, 17], researchers have designed powerful segmentation models [34, 7, 6, 8, 10] which are mostly equipped with an encoder-decoder architecture. These models have achieved success in various image do- mains, including medical image analysis, in particular or- gan and soft-tissue segmentation, which forms an important prerequisite of computer-assisted diagnosis [40, 41, 56, 54]. Medical images can appear in more than one phases, each of which corresponds to a specific way of data sam- * This work is done when this author was working in SJTU and JHU Cycle- GAN [57] Cy- CADA [19] UCDA [12] SIFA [5] PCN (ours) Image adaptation     Feature adaptation     Collaborative learning   Unknown label inference     Table 1: A comparison between our problem setting and that of previous approaches. Our approach stands out with the task of collaborative learning (see texts for details). pling and scanning. It has been well acknowledged that in- corporating multi-phase information improves visual recog- nition [42, 52]. Nevertheless, there have fewer studies on this problem. There are two possible reasons – one of them lies in the lack of multi-phase training data, and the other refers to the difficulty in aligning multi-phase data and dig- ging complementary information out from them. In this paper, we study this issue in the field of CT scans, for which we construct a large-scale dataset of 200 patients. For each case, two 3D volumes were collected from the ar- terial and venous phases, and the radiologists in our team manually annotated several abdominal targets, including or- gans and blood vessels. This is to say, in our dataset, each sample is composed of two paired images from arterial and venous phase, respectively. Note that the images scanned at the same position can be largely different, due to the dif- ference of radiation in scanning. Plus, although they are scanned from the same patient, the organs and vessels are not corresponded in pixel level due to their motion in the human body (see Figure 1). This causes huge difficulties in registrations [50, 2] between the two phases. Our goal is to train a model that leverages the information from both phases in a collaborative way and improves the segmenta- tion, while conventional approaches dealt with data in either phase, but missed the inter-phase connection. To model the inter-phase relation without the need of inter-phase registration, the problem refers to domain trans- 1 arXiv:1811.11814v3 [cs.CV] 12 Sep 2019