Car Detection by Fusion of HOG and Causal MRF SATISH MADHOGARIA Fraunhofer FKIE Wachtberg, Germany PAUL M. BAGGENSTOSS, Member, IEEE Naval Undersea Warfare Center Newport, RI, USA MAREK SCHIKORA WOLFGANG KOCH, Fellow, IEEE Fraunhofer FKIE Wachtberg, Germany DANIEL CREMERS Technical University of Munich Garching, Germany Detection of cars has a high variety of civil and military applications, e.g., transportation control, traffic monitoring, and surveillance. It forms an important aspect in the deployment of autonomous unmanned aerial systems in rescue or surveillance missions. In this paper, we present a two-stage algorithm for detecting automobiles in aerial digital images. In the first stage, a feature-based detection is performed, based on local histogram of oriented gradients and support vector machine classification. Next, a generative statistical model is used to generate a ranking for each patch. The ranking can be used as a measure of confidence or a threshold to eliminate those patches that are least likely to be an automobile. We analyze the results obtained from three different types of data sets. In various experiments, we present the performance improvement of this approach compared to a discriminative-only approach; the false alarm rate is reduced by a factor of 7 with only a 10% drop in the recall rate. Manuscript received March 14, 2012; revised December 17, 2012, May 20, 2012, October 31, 2013; released for publication November 4, 2013. DOI. No. 10.1109/TAES.2014.120141. Refereeing of this contribution was handled by H. Kwon. Authors’ addresses: S. Madhogaria, M. Schikora, W. Koch, Fraunhofer FKIE, Sensor Data and Fusion, Fraunhofer Str. 20, 53343 Wachtberg, Germany, E-mail: (satish.madhogaria@fkie.fraunhofer.de, marek.schikora@fkie.fraunhofer.de, wolfgang.koch@fkie. fraunhofer.de); P. M. Baggenstoss, Naval Undersea Warfare Center, Newport, RI (paul.m.baggenstoss@ieee.org); D. Cremers, Technical University of Munich, Department of Computer Science, Informatik 9, Bolzmannstrasse 3, 85748 Garching, Germany (daniel.cremer@in.tum.de). 0018-9251/15/$26.00 C  2015 IEEE I. INTRODUCTION Detecting objects in images is an interesting problem with many applications, like image retrieval and video surveillance. We have focused on detecting cars from noisy aerial images. Finding cars automatically is a hard task because the background in urban areas is highly complicated and cars tend to appear small with varying intensity or color. The appearance of the object within the observed scene also changes quite often depending on the flight altitude and camera orientation. Most classification approaches fall into one of two categories: generative classifiers, such as density estimation methods that seek to estimate a probability density function of the data, and discriminative methods, such as support vector machines (SVMs) that seek to find the best decision boundaries. These two general approaches have been compared, with the results usually being in favor of discriminative approaches [1, 2]. Rather than choosing one method, this paper aims to make use of the advantages of both generative and discriminative classifiers. Methods based on histogram of orientation gradients (HOG) have already proved their dominance in object recognition. However, when the object is very small in a complicated background, like an aerial view of a car in images from an unmanned aerial system (UAS), edge-based detection techniques (like HOG) may result in high number of false alarms. False detections are mainly due to the close resemblance of cars to various rectangular structures in the background, such as rooftops, windowpanes, markings on the highway, and so on. In our approach, we combine the HOG-based method with a novel causal Markov random field (MRF). We have chosen to combine these two classifiers because of the complementary information that they can provide. We combine them in a pipeline approach where the first stage (HOG-SVM) aims to achieve high recall rate with a relatively high false detection rate by relaxing the decision parameters of the discriminative classifier. The second stage (causal MRF) is designed to discard most of the false positives. We demonstrate the improved performance of the two-stage algorithm over the discriminative-only approach. We also analyze the performance of the combined method on standard data sets. Finally, with our sample aerial photographs, we show that the proposed method has the potential to be used for UAS-based surveillance. A. Related Work Various approaches have been proposed for vehicle detection in aerial images, including a neural-network-based hierarchical model for detection [3], use of gradient features to create a generic model and Bayesian network for classification [4], feature extraction comprising geometric and radiometric features and detection using a top-down matching approach [5], and an online AdaBoost approach [6]. HOG-based features have consistently outperformed in various object detection IEEE TRANSACTIONS ON AEROSPACE AND ELECTRONIC SYSTEMS VOL. 51, NO. 1 JANUARY 2015 575