A Semi-Automatic Traffic Sign Detection, Classification and Positioning System I.M. Creusen a,b , L. Hazelhoff a,b and P.H.N. De With a,b a Cyclomedia Technology, Achterweg 38, Waardenburg, The Netherlands; b Eindhoven University of Technology, Den Dolech 2, Eindhoven, The Netherlands ABSTRACT The availability of large-scale databases containing street-level panoramic images offers the possibility to per- form semi-automatic surveying of real-world objects such as traffic signs. These inventories can be performed significantly more efficiently than using conventional methods. Governmental agencies are interested in these inventories for maintenance and safety reasons. This paper introduces a complete semi-automatic traffic sign in- ventory system. The system consists of several components. First, a detection algorithm locates the 2D position of the traffic signs in the panoramic images. Second, a classification algorithm is used to identify the traffic sign. Third, the 3D position of the traffic sign is calculated using the GPS position of the photographs. Finally, the results are listed in a table for quick inspection and are also visualized in a web browser. 1. INTRODUCTION Recently several parties have commenced collecting street-level panoramic photographs on a country-wide scale. Examples are Google’s street view * , Microsoft streetside * , Mapjack * , Everyscape * , Daum’s Road View * and Cyclomedia’s Globespotter * . These large-scale collections offer the possibility to inventory real-world objects in a much more efficient way. Previously, the creation of inventories was performed completely manually by driving around and annotating every object by hand. By applying computer vision algorithms to these large collections of photographs, the work can be performed more efficiently, requiring less manual interaction. Governmental agencies maintain many types of street furniture such as roads, street lights, trash cans, traffic signs and many others. Traffic signs, which are the focus of this paper, frequently become damaged by collisions, are stolen, become dirty or become occluded by trees. In addition to these external factors that can influence the objects, traffic safety can be often improved by adding additional signs or removing unnecessary signs. Up- to-date inventories are useful for this maintenance task, as they can aid in the detection and prevention of the previously mentioned problems. Similar to the findings by Frome et al., 1 we have found that the performance of state-of-the-art algorithms for detection and classification of traffic signs is insufficient for a fully automated inventorying process. Therefore a semi-automatic approach was taken, to ensure the quality of the resulting inventory is sufficient. The proposed inventorying system consists of several parts. First, a detection algorithm is used to locate the traffic signs within the panoramic photographs. Second, the resulting detections are classified by a classification algorithm, to determine the traffic sign type. After manual verification of these results, the positioning algorithm combines the detections of nearby panoramas to calculate the 3D coordinates of the traffic signs by triangulation. Finally, the traffic signs are visualized in a flash application running in a web browser. The traffic signs are projected on the map, as well as in the photos themselves, allowing for a good overview. In literature, these parts are often considered as independent algorithms, whereas in this paper, a joint design of the complete framework is elaborated. The rest of the paper is structured as follows. We begin by introducing the dataset and its difficulties in Section 2. Section 3 shows an overview of the system and its components. In Section 4 we describe the Detection algorithm in more detail. Finally the classification algorithm is presented in Section 5. followed by the 3D positioning algorithm in Section 6. Section 7 covers practical application of the system and current limitations. Finally, the conclusion can be found in Section 8. * www.google.com/streetview, www.microsoft.com/maps/streetside.aspx, www.mapjack.com, www.everyscape.com, local.daum.net/map, www.globespotter.eu