CLOUD-GAN : CLOUD REMOVAL FOR SENTINEL-2 IMAGERY USING GENERATIVE ADVERSARIAL NETWORKS Praveer Singh Nikos Komodakis École des Ponts ParisTech & Université Paris Est, France Praveer.Singh@enpc.fr Remote Sensing (RS) imagery is pivotal for varied challenging tasks such as recognizing footprints of buildings [1], detecting changes in temporarily apart scenes [2] or semantic segmentation in aerial scenes [3]. Such images are often plagued by films of clouds that partially or completely obstruct the scene. This can be quite annoying for RS experts, especially while observing a city like Paris which witnesses cloudy weather for a major part of the year. Thus, it clearly necessitates the requirement for an automatic technique that detects and removes the cloudy regions in a scene and replaces them with a neat in-painting of the underlying scene. Predicting a scene beneath a cloud is an under-constrained problem and unless we have some prior information, it is largely quite complex to replace clouds with correct underlying details. A way out has been by using multi-temporal images of the same region as done by [4] through a Multi-Temporal Dictionary Learning. [5] used Synthetic Aperture Radar (SAR) Imagery owing to the fact that it can easily penetrate through the clouds. However, SAR imagery is difficult to interpret and have a lower spatial resolution compared to RGB imagery. Additionally, [6, 7] studied the thin cloud removal problem in the literature. However, most of these approaches were based on conventional hand crafted methods and are limited in terms of performance. Generative Adversarial Networks (GANs) [8] have gained immense popularity owing to their remarkable capability in mo- deling the mapping function between input and output images belonging to target domains. Using an adversarial loss, the GAN’s can be trained to produce fake images which are indistinguishable from the real images of target domain. [9] used McGANs to predict cloud-free RGB images as well as cloud masks from the input cloudy image. The authors trained the model using pair of cloud-free image and synthetically produced cloudy images (by adding Perlin noise to the original RGB images). Additionally, they also utilize Near-Infrared (NIR) imagery, which is closer to visible range and possess partial cloud penetration capabilities. However, such kind of synthetic clouds produced using Perlin noise, are not realistic and si- gnificantly different from actual clouds seen in visible light images. Nevertheless, composing a dataset of real clouds and their cloud-free counterparts is quite a herculean task. We overcome this hurdle, by improvising upon a novel technique, which together with the adversarial loss from traditional GANs, employs a more recent cycle consistent loss [10], to convert thin cloudy images to cloud-free RGB images. Having a cycle consistency loss constrains the problem, such that if an image is transformed from input domain to target and then back to the input domain, it should look alike to the original image. An additional advantage of our method is that it absolves us from the requirement of an explicit paired cloudy/cloud-free dataset. Moreover, our methodology doesn’t require any sort of cloud-penetration sources of imagery such as SAR or NIR. We simply utilize visible range imagery from a fairly new open source dataset (Sentinel-2) to report impressive results clearly showcasing the efficacy of our results. We also report quantitative results on synthetically generated cloudy images, showcasing a significant improvement in PSNR values. Références [1] Emmanuel Maggiori, Yuliya Tarabalka, Guillaume Charpiat, and Pierre Alliez, “Can semantic labeling methods gene- ralize to any city ? the inria aerial image labeling benchmark,” in IEEE International Geoscience and Remote Sensing Symposium (IGARSS). IEEE, 2017. [2] H. Lyu, H. Lu, and L. Mou, “Learning a transferable change rule from a recurrent neural network for land cover change detection,” Remote Sensing, vol. 8, no. 6, 2016. [3] N. Audebert, B. Le Saux, and S. Lefèvre, “How useful is region-based classification of remote sensing images in a deep learning framework ?,” in IGARSS, July 2016, pp. 5091–5094. [4] M. Xu, X. Jia, M. Pickering, and A. J. Plaza, “Cloud removal based on sparse representation via multitemporal dictionary learning,” IEEE Transactions on Geoscience and Remote Sensing, vol. 54, no. 5, pp. 2998–3006, May 2016.