Abstract— For autonomous robots equipped with a camera, terrain classification is essential in finding a safe pathway to a destination. Terrain classification is based on learning, but the amount of data cannot be infinite. This paper presents a self-supervised classification approach to enable a robot to learn the visual appearance of terrain classes in various outdoor environments by observing moving objects, such as humans and vehicles, and to learn about the terrain, based on their paths of movement. We verified the performance of our proposed method experimentally and compared the results with those obtained using supervised classification. The difference in error rates between self-supervised and supervised methods was about 0–11%. I. INTRODUCTION HEN an autonomous robot needs to reach a destination, the most important and basic process before moving is assessing the safety of the surrounding environment and finding a safe path of movement. This process could incorporate GPS and mapping technology, but a GPS system is not accurate enough to identify an exact position, and maps do not include all objects, especially moving objects. Thus, the ability to recognize the surrounding environment is essential for a moving robot. One recognition method is terrain classification, in which a robot uses sensor responses to recognize the surrounding environment and determine safe possible pathways. During the past year, several studies have proposed methods for terrain classification. One study developed a method that involved searching for possible obstacles using a stereo camera, eliminating candidates based on texture and color clues, and then modeling terrain after obstacles had been defined [1]. Another study focused on avoiding trees in a forest; it used a stereo camera to recognize trees and classify terrain to find a safe pathway [2]. Other studies used a vibrating sensor to classify terrain that had already been traveled, based on various vibration frequencies [3] [4]. However, these techniques only work in specific environments. Robots need to be able to learn about unknown terrain. Some studies have focused on supervised learning that requires human intervention when a robot reaches an unknown area. Due to the limitations of supervised learning, Donghui Song is with the Department of Intelligent Robot Engineering, Hanyang University, Korea. donghi315@gmail.com. Chuho Yi is with the Division of Electrical and Computer Engineering, Hanyang University, Korea. $OO FRUUHVSRQGHQFHV VKRXOG EH DGGUHVVHG WR & <L d1uck@hanyang.ac.kr Il Hong Suh is with the College of Information and Communications, Hanyang University, Korea. ihsuh@hanyang.ac.kr Byung-Uk Choi is with the College of Information and Communications, Hanyang University, Korea. buchoi@hanyang.ac.kr Fig. 1. Outdoor environment (black text: moving objects, white text: various terrain regions) many researchers are now working on self-supervised or unsupervised techniques, in which a robot can learn about an environment on its own, without any human supervision. One recent study developed a technique in which a robot can calculate the depth of a ground plane using a depth map generated by a stereo camera, and can classify and learn about the ground and obstacles within 12 m. Based on these data, it can recognize very distant regions, as far as 30–40 m [5]. Another study developed an unsupervised learning method that deletes incorrect detections about a wide variety of terrain types (e.g., trees, rocks, tall grass, bushes, and logs) while the robot navigates and collects data [6]. Other studies involved self-supervised classification using two classifiers: an offline classifier that used vibration frequencies or laser to provide the other classifier, an online and visual classifier, with labels for various observed terrains. This allowed the visual classifier to learn about, and recognize, new environments [7][19]. However, some of these methods require more than one sensor; some use stereo cameras or vibrating sensors with monocular cameras, and most assume either that the robot is facing a flat plane through which it can navigate or that the robot will learn about the terrain after it navigates through it. In this study, we developed a method of self-supervised learning using moving objects. The robot observes human and vehicles as moving objects and determines the paths they take. These paths are used to generate terrain data, which the robot learns. The method was inspired by how humans act using indirect experiences: they can learn from the behavior of others without actually performing a task. One advantage of the proposed method is that the robot can obtain knowledge about the surrounding environment based solely on observation. It does not require that the robot move, and is thus less risky than other methods. Figure 1 shows one of our experiment’s environments which indicate moving objects and various types of terrain. Self-Supervised Terrain Classification Based on Moving Objects Using Monocular Camera Donghui Song, Chuho Yi, Il Hong Suh, and Byung-Uk Choi W 978-1-4577-2138-0/11/$26.00 © 2011 IEEE 527 Proceedings of the 2011 IEEE International Conference on Robotics and Biomimetics December 7-11, 2011, Phuket, Thailand