Robotics and Autonomous Systems 131 (2020) 103578 Contents lists available at ScienceDirect Robotics and Autonomous Systems journal homepage: www.elsevier.com/locate/robot Real-time topological localization using structured-view ConvNet with expectation rules and training renewal Chih-Hung G. Li a,* , Yi-Feng Hong a , Po-Kai Hsu a , Thavida Maneewarn b a College of Mechanical and Electrical Engineering, National Taipei University of Technology, Taipei 10608, Taiwan b Institute of Field Robotics, King Mongkut’s University of Technology Thonburi, Bangkok 10140, Thailand article info Article history: Received 28 July 2019 Received in revised form 28 February 2020 Accepted 25 May 2020 Available online 10 June 2020 abstract Mobile service robots possess high potential of providing numerous assistances in the working areas. In an attempt to develop a mobile service robot which is dynamically balanced for faster movement and taller manipulation capability, we designed and prototyped J4.alpha, which is intended for swift navigation and nimble manipulation. Previously, we devised a pure visual method based on a supervised deep learning model for real-time recognition of nodal locations. Four low-resolution RGB cameras are installed around J4.alpha to capture the surrounding visual features for training and detection. As the method is developed for ease of implementation, fast real-time application, accurate detection, and low cost, we further improve the accuracy and the practicality of the method in this study. Specifically, a set of expectation rules are introduced to reject outlier detections, and a scheme of training renewal is devised to effectively react to environmental modifications. In our previous tests, precision and recall rates of the location coordinate detection by the ConvNet models were generally between 0.78 and 0.91; by introducing the expectation rules, precision and recall are improved by approximately 10%. A large scale field test is also carried out here for both corridor and factory scenarios; the performance of the proposed method was tested for detection accuracy and verified for 2 m and 0.5 m nodal intervals. The scheme of training renewal designed for capturing and reflecting environmental modifications was also proved to be effective. © 2020 Elsevier B.V. All rights reserved. 1. Introduction As part of our effort toward developing a mobile service robot referred as J4.alpha, a visual topological localization system is designed and implemented. As shown in Fig. 1, J4.alpha is a two- wheeled self-balanced mobile robot equipped with two 5-DOF manipulators. The development project aims to build a proto- type of the mobile robot for practical use in hospitals, facto- ries, campuses, offices, warehouses, etc. The autonomous/semi- autonomous system is expected to perform tasks such as environ- ment surveillance, object pick-and-place, equipment operation, wireless communication, etc. At a height of 1.35 m, J4.alpha is tall enough to conduct meaningful interactions with human beings; the compact two-wheeled base lets it easily adapt to and be accepted in human-centric environments. So far, a fully autonomous mobile robot capable of exploring unknown envi- ronments and performing real-time and accurate SLAM — simul- taneous localization and mapping is still difficult to achieve. It is much easier to narrow down the scope and just familiarize the mobile robot with the environments that it works in. And * Corresponding author. E-mail address: cl4e@ntut.edu.tw (C.-H.G. Li). in practice, often we only need the robot to provide services between fixed locations such as the warehouse and the CNC machines or wards in the hospital and the medical stations. A service robot which is capable of navigating along the predeter- mined paths between designated locations can be quite useful. A common strategy adopted in industry is by using magnetic tapes placed on the floor for provision of directional and distance guidance for the robot. Whereas the method has been proved to be sufficiently effective and reliable, there are intrinsic drawbacks with it. First of all, taping the floor is a construction work requir- ing time and funding. The action can be essentially intrusive to the human-centric environments and may not be welcomed in certain circumstances. Secondly, the magnetic tape can be subject to contamination, wear, or other interfering means and loses the reliability over time. Recently, visual localization and place recognition has been progressing rapidly; advancement is further propelled by deep learning. To devise the navigation system for J4.alpha, we di- vide the system into two parts, a localization system through visual topological localization and a self-driving system through visual obstacle avoidance and direction planning. In this paper, we report the visual topological localization system developed for J4.alpha. Specifically, a global feature-training approach was https://doi.org/10.1016/j.robot.2020.103578 0921-8890/© 2020 Elsevier B.V. All rights reserved.