Pedestrian Detection: The Elephant In The Room Irtiza Hasan 1 , Shengcai Liao ⋆1 , Jinpeng Li 1 , Saad Ullah Akram 2 , and Ling Shao 1 1 Inception Institute of Artificial Intelligence (IIAI), Abu Dhabi, UAE {irtiza.hasan,shengcai.liao,jinpeng.li,ling.shao}@inceptioniai.org 2 Department of Computer Science, Aalto University, Finland saad.akram@aalto.fi Abstract. Pedestrian detection is used in many vision based applica- tions ranging from video surveillance to autonomous driving. Despite achieving high performance, it is still largely unknown how well ex- isting detectors generalize to unseen data. To this end, we conduct a comprehensive study in this paper, using a general principle of direct cross-dataset evaluation. Through this study, we find that existing state- of-the-art pedestrian detectors generalize poorly from one dataset to an- other. We demonstrate that there are two reasons for this trend. Firstly, they over-fit on popular datasets in a traditional single-dataset training and test pipeline. Secondly, the training source is generally not dense in pedestrians and diverse in scenarios. Accordingly, through experiments we find that a general purpose object detector works better in direct cross-dataset evaluation compared with state-of-the-art pedestrian detec- tors and we illustrate that diverse and dense datasets, collected by crawl- ing the web, serve to be an efficient source of pre-training for pedestrian detection. Furthermore, we find that a progressive training pipeline works good for autonomous driving oriented detector. We improve upon previ- ous state-of-the-art on reasonable /heavy subsets of CityPersons dataset by 1.3%/1.7% and on Caltech by 1.8%/14.9% in terms of log average miss rate (MR -2 ) points without any fine-tuning on the test set. Detector trained through proposed pipeline achieves top rank at the leaderborads of CityPersons [42] and ECP [4]. Code and models will be available at https://github.com/hasanirtiza/Pedestron. Keywords: Pedestrian Detection, Autonomous Driving, Video Surveil- lance, Transfer Learning and Robotics 1 Introduction Pedestrian detection is a very actively researched task in the computer vision community, both in academia and industry. It has applications in many different domains, including robotics and autonomous vehicles, entertainment, and smart ⋆ Corresponding author arXiv:2003.08799v2 [cs.CV] 22 Mar 2020