An approach for identifying classifiable regions of an image captured by autonomous robots in structural environments Andrew Wing Keung To n , Gavin Paul, Dikai Liu Centre for Autonomous Systems, University of Technology, Sydney, Australia article info Article history: Received 31 March 2014 Received in revised form 17 July 2015 Accepted 23 July 2015 Keywords: Surface-type classification RGB-D Likelihood Texture features abstract When an autonomous robot is deployed in a structural environment to visually inspect surfaces, the capture conditions of images (e.g. camera's viewing distance and angle to surfaces) may vary due to un- ideal robot poses selected to position the camera in a collision-free manner. Given that surface inspection is conducted by using a classifier trained with surface samples captured with limited changes to the viewing distance and angle, the inspection performance can be affected if the capture conditions are changed. This paper presents an approach to calculate a value that represents the likelihood of a pixel being classifiable by a classifier trained with a limited dataset. The likelihood value is calculated for each pixel in an image to form a likelihood map that can be used to identify classifiable regions of the image. The information necessary for calculating the likelihood values is obtained by collecting additional depth data that maps to each pixel in an image (collectively referred to as a RGB-D image). Experiments to test the approach are conducted in a laboratory environment using a RGB-D sensor package mounted onto the end-effector of a robot manipulator. A naive Bayes classifier trained with texture features extracted from Gray Level Co-occurrence Matrices is used to demonstrate the effect of image capture conditions on surface classification accuracy. Experimental results show that the classifiable regions identified using a likelihood map are up to 99.0% accurate, and the identified region has up to 19.9% higher classification accuracy when compared against the overall accuracy of the same image. & 2015 Elsevier Ltd. All rights reserved. 1. Introduction The typical manufacturing factory setting provides a well- structured and predictable environment that is suitable for im- plementing pre-planned routines on a robot to perform repetitive tasks. However, pre-planned routines are impractical for a field robot due to the environment changing over time and/or the robot being freely moved within the environment. One such scenario is steel bridge maintenance conducted by autonomous robots [1,2], as shown in Fig. 1. In steel bridge maintenance, a mobile robot is moved to various sections of a bridge to conduct grit-blasting for the removal of rust and old paint from steel surfaces in preparation for repainting. In order for the robot to grit-blast autonomously in each position, an up-to-date geometric map of the surrounding environment is provided to the robot such that a plan for grit-stream trajectory and robot movements can be newly generated. At present, there are well-developed approaches for a robot to explore and build an update geometric map of an environment using a depth sensor mounted on the robot's end-effector [3,4]. Provided with a geo- metric map of the environment, a robot can only autonomously grit-blast all the surfaces without the capability to target specific surface areas based on surface-type/conditions (e.g. mildly rusted and heavily rusted). For a robot to be capable of selectively grit-blasting specific surface areas, it must also explore and inspect the surface's con- dition. One possible approach to this is to mount a vision camera to the robot's end-effector and capture images during (1) pre-grit- blasting for identifying specific surface areas to grit-blast based on rust grading, and (2) post-grit-blasting for assessing whether the required steel cleanliness has been achieved or re-blasting is ne- cessary. A robot can inspect the surfaces in the captured images by using a classifier trained with surface samples from a visual in- spection standard such as the rust grading and steel cleanliness visual metrics provided in BS EN ISO 8501-1 [5]. In this way, in- formation about the surface's condition can be produced that will enable a robot to intelligently (re)grit-blast specific surface areas on a bridge. A review of vision-based classification approaches shows that colour and/or texture features can be extracted to accurately dis- tinguish between various surface-types (surface appearance of Contents lists available at ScienceDirect journal homepage: www.elsevier.com/locate/rcim Robotics and Computer-Integrated Manufacturing http://dx.doi.org/10.1016/j.rcim.2015.07.003 0736-5845/& 2015 Elsevier Ltd. All rights reserved. n Corresponding author. E-mail addresses: Andrew.To-1@uts.edu.au (A.W.K. To), Gavin.Paul-1@uts.edu.au (G. Paul), Dikai.Liu@uts.edu.au (D. Liu). Robotics and Computer-Integrated Manufacturing 37 (2016) 90–102