SEMANTIC SEGMENTATION AND DIFFERENCE EXTRACTION VIA TIME SERIES AERIAL VIDEO CAMERA AND ITS APPLICATION S.N.K. Amit a, *, S. Saito a , S.Sasaki b,c , Y. Kiyoki b , Y. Aoki a a Keio University, Graduate School of Science and Technology, 3-14-1 Hiyoshi, Kohoku-ku, Yokohama, Kanagawa, 223-8522 Japan – {sitinork, ssaito} @aoki-medialab.org, aoki@elec.keio.ac.jp b Keio University, Graduate School of Media and Governance, 5322 Endo, Fujisawa-shi, Kanagawa, 252-0882 Japan – shiori.sasaki@gmail.com, kiyoki@sfc.keio.ac.jp c Chulalongkorn University, 254 Phayathai Road, Pathumwan, Bangkok 10330, Thailand – shiori.sasaki@gmail.com KEY WORDS: difference extraction, semantic segmentation, aerial images, convolution neural networks ABSTRACT: Google earth with high-resolution imagery basically takes months to process new images before online updates. It is a time consuming and slow process especially for post-disaster application. The objective of this research is to develop a fast and effective method of updating maps by detecting local differences occurred over different time series; where only region with differences will be updated. In our system, aerial images from Massachusetts’s road and building open datasets, Saitama district datasets are used as input images. Semantic segmentation is then applied to input images. Semantic segmentation is a pixel-wise classification of images by implementing deep neural network technique. Deep neural network technique is implemented due to being not only efficient in learning highly discriminative image features such as road, buildings etc., but also partially robust to incomplete and poorly registered target maps. Then, aerial images which contain semantic information are stored as database in 5D world map is set as ground truth images. This system is developed to visualise multimedia data in 5 dimensions; 3 dimensions as spatial dimensions, 1 dimension as temporal dimension, and 1 dimension as degenerated dimensions of semantic and colour combination dimension. Next, ground truth images chosen from database in 5D world map and a new aerial image with same spatial information but different time series are compared via difference extraction method. The map will only update where local changes had occurred. Hence, map updating will be cheaper, faster and more effective especially post-disaster application, by leaving unchanged region and only update changed region. * Corresponding author. This is useful to know for communication with the appropriate person in cases with more than one author. 1. INTRODUCTION 1.1 Background Many people misunderstood satellite imagery on Google Earth is real time imagery. Actually once the imagery is taken, commercial provider like GeoEye will takes time to process the data before providing it to the customer such as Google. Google has to evaluate the new imagery with the current imagery to determine whether the new one is better than the current. One an image is selected, it has to be processed into the format and coordinate system of Google Earth’s databases. Then it has to undergo a quality control process and fed into a processing system before it gets distributed to the live Google Earth database server. Hence, imagery on Google Earth is usually more than 6 months old. And updates only happen about once every 60 days. Each updates covers a very small portion of the globe. For example, only 4-6 cities are updated in United States, or maybe just a single state, and other countries receive similar minor updates. As a general rule, Google tries to keep every area updated to within around three years old. The exception to that is when there is a major disaster, such as earthquake in Haiti, or tsunami in Fukushima, Japan. During and after an event like that, Google posts fresh imagery as quickly as possible to emergency workers and concerned residents of the affected areas. However, this is still a slow process especially for big disaster that occurred, even 1 minute of tsunami, everything on Fukushima prefecture, Japan is gone. Furthermore, it will be helpless situation if the emergency workers and other related people just have an old imagery, which is not up-to-date, and they are not able to get near to the affected disaster area due to dangerous radio waves. 1.2 Related Works In the past few years, Urabe K and team members has done a research on detection of road blockage in mountainous area using the combination of satellite images and aerial images before and after a disaster occurred. They used RGB satellite images and infrared satellite images before a disaster occurred, to compare with aerial images after a disaster occurred with the help of digital elevation model by using simple difference extraction method to detect blocked road blockage. This method is able to detect road blockage in mountainous area precisely up to 80%, however it has a lot of premises such as limited to sunny day, imagery taken only in the morning and road is clearly seen from satellite. Besides, this method is only suitable for road detection on mountainous area, which is limited and unable to be used globally for other geographical changes. 1.3 Research Objective Hence, we would like to propose a new framework to detect geographical changes to keep Google imagery up-to-date by using image processing, and able to process the latest imagery within a few hours especially after a disaster occurred. The International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences, Volume XL-7/W3, 2015 36th International Symposium on Remote Sensing of Environment, 11–15 May 2015, Berlin, Germany This contribution has been peer-reviewed. doi:10.5194/isprsarchives-XL-7-W3-1119-2015 1119