Radar and Camera Sensor Fusion with ROS for Autonomous Driving Rahul Kumar Radar Systems and Sensor Fusion, Flux Auto Pvt. Ltd. resorahul@gmail.com Sujay Jayashankar Computer Vision and Deep Learning, Flux Auto Pvt. Ltd. sujay.jayashankar@fluxauto.xyz Abstract—As we are diving deeper into the world of Autonomous Driving, we need a more reliable and robust system which can sense the surrounding of a vehicle in all- weather conditions and at any time of the day. Apart from this, we also need a system which is highly accurate in terms of measuring the parameters like distance and velocity of the objects in its field of view. Keeping these factors in mind we have proposed a robust model to fuse camera and radar data, which is an economical choice with a proven performance. This model is implemented using Robot Operating System (ROS) environment, which is an Open source middleware for Autonomous Driving Application. Keywords—Sensor Fusion, FMCW Radar, ROS, PointCloud, Camera Calibration I. INTRODUCTION Autonomous driving being a very sophisticated problem statement, we need highly robust and reliable systems which can perform in all weather conditions round the clock. The sensor technologies available to us have their own pros and cons. Some work better than others in different weather and light conditions. One such example is the mmWave Radars which are becoming a vital part for the ADAS and autonomous driving applications [1]-[5]. The conventional approach to autonomous driving has changed significantly, instead of using only one type of sensors be it Vision, Radar or Lidar, now we are using a combination of two sensors or all of them together to make a more robust system. We can clearly see the effect on the performance of these sensors under different conditions as well as the feasibility in terms of cost in Fig. 1 below. Figure 1. Comparison of Camera, Radar and Lidar sensors The idea behind the fusion of radar and camera sensors is that radar gives us an accurate measurement of the distance, radial velocity, longitudinal velocity, azimuth and the elevation of the objects which are in its FOV. A monocular camera provides an output which is easily perceivable and is an ideal input for detecting and classifying objects. For autonomous driving application the sensors range must be at least 150 to 200 metres. The fusion of camera and radar will allow us to detect and classify the objects at this range. Sensor fusion is an approach with which you can avoid the dependency of the autonomy stack on one sensor and can make the system more robust with the help of the feedback from different sensors. Even if a part of the autonomy stack starts malfunctioning the remaining sensors will keep the system running. The radar was interfaced using the ROS (Robot Operating System) which is an open source middleware. The measurements of the objects detected by the radar were taken in PointCloud format. Further, the 3D points from the radar were converted to the image pixel coordinates. Then the pixel coordinates from the radar were further mapped with the objects detected in the camera frame using a classifier. The objects were then tracked, both in the camera as well as the radar frame. The current technology of automotive radars gives us very precise measurements of parameters like distance, velocity, azimuth and elevation of the objects in its FOV, be it moving or stationary objects. But the limitation being, it cannot tell us exactly what the object is. II. RELATED WORK In the paper "Data fusion of radar and stereo vision for detection and tracking of moving objects,” [6], a method for combining the information from stereo vision cameras and a monopulse FMCW radar for the detection and tracking of moving objects is discussed. The system detects moving objects for both sensor systems individually, before fusing the information in order to get more accurate detections. In the paper “Lidar and Camera Detection Fusion in a Real- Time Industrial Multi-Sensor Avoidance system” by Pan Wei, Lucas Cagle, Tasmia Reza, John Ball, James Gafford (30 May 2018) [7], proposed the Fusion of Lidar and camera for detecting specific objects in the environment and avoid collision with it and noticed a major drawback with using Lidar as false positive reflections are recorded which result in incorrect detections. Similarly, to implement in an uncontrolled environment it’s impractical as there is no certainty on the conditions which will cause false positive reflections. The paper [8], presented an algorithm based on the most cited and common clustering algorithm: DBSCAN. The algorithm was modified to deal with the non- equidistant sampling density and clutter of radar data while maintaining all its prior advantages. The algorithm outperforms DBSCAN in terms of speed by using the knowledge of the sampling density of the sensor (increase of app. 40-70%). There are different algorithms and techniques used for sensing and tracking of targets using radars [9]-[11]. Also, the paper [12], compares two algorithms for vehicle tracking using radar and camera data. Inspired by the heuristic fusion with adaptive gating, we tried implementing something similar using a centroid tracker. The paper "Extending Reliability of mmWave Radar Tracking and Detection via Fusion With Camera," [13], takes into consideration the error bounds of the two different coordinate systems from the heterogeneous sensors, and a fusion- 2019 Fifth International Conference on Image Information Processing (ICIIP) 978-1-7281-0899-5/19/$31.00 ©2019 IEEE 568