Contents lists available at ScienceDirect ISPRS Journal of Photogrammetry and Remote Sensing journal homepage: www.elsevier.com/locate/isprsjprs Developmentandevaluationofadeeplearningmodelforreal-timeground vehicle semantic segmentation from UAV-based thermal infrared imagery Mehdi Khoshboresh Masouleh, Reza Shah-Hosseini ⁎ School of Surveying and Geospatial Eng., College of Eng., University of Tehran, Tehran, Iran ARTICLEINFO Keywords: UAV-based thermal infrared imagery Ground vehicle Semantic segmentation Deep learning Gaussian-Bernoulli Restricted Boltzmann Machine ABSTRACT Real-time unmanned aerial vehicles (UAVs)-based thermal infrared images processing, due to high spatial re- solution and knowledge of the various infrared radiant energy level distribution of solid bodies, has important applications such as monitoring and control of the various phenomena in different natural situations. One of these applications is monitoring the ground vehicles in cities by using detection or semantic segmentation of them in the thermal images. In this research, our purpose is to improve the performance of deep learning combined model by using Gaussian-Bernoulli Restricted Boltzmann Machine (GB-RBM) specifications for the segmentationofthegroundvehiclesfromUAV-basedthermalinfraredimagery.Theproposedmodelisstudied in three steps. First, designing the proposed model by using an encoder-decoder structure and addition of ex- tracted features from convolutional layers and restricted Boltzmann machine in the network. Second, the im- plementation of the research goals on four sets of UAV-based thermal infrared imagery named NPU_CS_UAV_IR_DATA that was collected from some streets of China by using FLIR TAU2 thermal infrared sensorin2017.Finally,analyzingtheperformanceoftheproposedmodelbyusingfivestate-of-the-artmodelsin semanticsegmentation.Theresultsevaluatedtheperformanceoftheproposedmodelasarobustmodelwiththe averageprecisionandaverageprocessingtimeofapproximately0.97,and19.73sforalldatasets,respectively. 1. Introduction Offering a purposeful and scientific strategy for monitoring the naturalandartificialelementsintheenvironmentisalwaysconsidered as an important issue in the plans of decision making of managers in every country (Famiyeh et al., 2018; Grekousis, 2018). Many of the researcheshaveusedthelowspatialresolutionremotesensingdatafor the monitoring of the various applications such as earthquake crisis management, flood monitoring, thermal map and soil moisture map production(Fisheretal.,2017).Butanalyzingtheremotesensingdata withlowspatialresolutionwon’thavetherequiredcapabilitiestomake important decisions, and most of the hazards and urban managers’ needs are in the local scale and with the awareness of details of the various phenomena (Restas, 2015). Nowadays, unmanned aerial ve- hicles (UAVs) imaging is the proper solution for monitoring different phenomenainthelocalscale(Arnoldetal.,2012;Helgesenetal.,2019; Yang and Chen, 2015). On the other hand, the extension of the UAV imaging utilization in the military and non-military applications and proposingapracticalmethodtoimprovethereal-timeimageprocessing with data redundancy prevention and high speed in the generation of spatial information is always welcomed (Konoplich et al., 2016). ThesemanticsegmentationofthegroundvehiclesfromUAV-based thermal infrared imagery has important applications in traffic mon- itoring (Karimi Nejadasl et al., 2006; Reinartz et al., 2006), analyzing the in-town and suburban accidents (Apeltauer et al., 2015), helping thecontrolofunmannedvehicles(Badshahetal.,2018;QinandZhang, 2018), managing the activity of ground vehicles in construction work- shops(Leeetal.,2017),vehiclestypeidentificationinaregioninorder to prepare an accurate statistics of car market (Zhong et al., 2017), analyzing the trespassing of vehicle in forbidden area (Kellenberger etal.,2018),managingtheopenspacesparkinglotsandidentification oflostvehiclesduetonaturalorabnormalaccidents(Zhouetal.,2017). Mostoftheusualmethodsforsemanticsegmentationoftheimagesare with the purpose of detection of vehicle-based on the ground imaging and with ground platforms. The most important problems in these methods are limited coverage of these cameras and control power de- crease in proper dimensions. Therefore, UAV imaging utilization is considered as a proper solution with low cost to monitor the local elements. The purpose of vehicle detection from UAV images is to locate the vehicleontheimagebyusingabox(alsoknownasboundingbox)that with the change in the box size, the location of the object will be https://doi.org/10.1016/j.isprsjprs.2019.07.009 Received10January2019;Receivedinrevisedform20July2019;Accepted20July2019 ⁎ Corresponding author. E-mail address: rshahosseini@ut.ac.ir (R. Shah-Hosseini). ISPRS Journal of Photogrammetry and Remote Sensing 155 (2019) 172–186 0924-2716/ © 2019 International Society for Photogrammetry and Remote Sensing, Inc. (ISPRS). Published by Elsevier B.V. All rights reserved. T