Robotics and Autonomous Systems 47 (2004) 109–116 Teaching and learning of robot tasks via observation of human performance Rüdiger Dillmann Institute of Computer Design and Fault Tolerance, Universität Karlsruhe (TH), P.O. Box 6980, Karlsruhe D-76128, Germany Abstract Within this paper, an approach for teaching a humanoid robot is presented that will enable the robot to learn typical tasks required in everyday household environments. Our approach, called Programming by Demonstration, which is implemented and successfully used in our institute to teach a robot system is presented. Firstly, we concentrate on an analysis of human actions and action sequences that can be identified when watching a human demonstrator. Secondly, sensor aid systems are introduced which augment the robot’s perception capabilities while watching a human’s demonstration and the robot’s execution of tasks respectively. The main focus is then layed on the knowledge representation in order to be able to abstract the problem solution strategies and to transfer them onto the robot system. © 2004 Elsevier B.V. All rights reserved. Keywords: Robot tasks; Programming by Demonstration; One-Shot-Learning 1. Introduction Service robots, which are supposed to assist hu- mans in their daily household work, must be adaptable and flexible. They must be adaptable on the one hand to the individual requirements and needs and on the other hand to the user’s environment. As each house- hold environment is different depending on its geomet- ric setup, its provided functionality and the requested tasks to be performed by the robot, ready-made pro- grams will not be able to cope with such environments. So, flexibility will surely be one of the most important requirements for the robot’s design. It must be able to navigate through a changing environment, to adapt its recognition abilities to a particular scenery and to manipulate a wide range of objects. Furthermore, it is extremely important for the user to easily adapt the E-mail address: dillmann@ira.uka.de (R. Dillmann). system to his needs, i.e. to teach the system what to do and how to do it. We think that the only approach that satisfies the latter condition are systems that automat- ically acquire relevant information for task execution by observation and multi-modal dialogues. Learning of action sequences or operation plans is an abstract problem, which supposes a modeling of cognitive skills. When learning complex action sequences, basic manipulation skills or controlling techniques are not investigated. In fact, the aim is to generate an abstract description of the demonstration reflecting the user’s intention and modeling the prob- lem solution as optimal as possible. Robot indepen- dence is an important issue because it would allow to exchange of robot programs between robots with dif- ferent kinematics. Likewise, a given problem has to be suitably generalized, for example, distinguishing parameters specific for the particular demonstration and parameters specific for the problem concept. Both 0921-8890/$ – see front matter © 2004 Elsevier B.V. All rights reserved. doi:10.1016/j.robot.2004.03.005