Affordance-Based Reasoning in Robot Task Planning Iman Awaad and Gerhard K. Kraetzschmar Bonn-Rhein-Sieg University and B-IT Center Grantham-Allee 20 53757 Sankt Augustin, Germany Joachim Hertzberg Osnabr¨ uck University and DFKI RIC Osnabr ¨ uck Branch Albrechtstrasse 28 49076 Osnabr ¨ uck, Germany Abstract Humans are able to come up with plans to achieve their goals, and to adapt these plans to changes in their environment, find- ing fixes, alternatives and taking advantages of opportunities without much deliberation. For example, they may use a tea kettle instead of a watering can to water the plants, or a mug instead of a glass to serve water. Despite decades of research, artificial agents are not as robust or as flexible. In this work, we introduce three reasoning phases that use affordances to enable such robustness and flexibility in robot task planning. The first phase generates a focused planning problem. The second phase expands the domain where necessary while the third and final reasoning phase uses affordances during plan execution and monitoring. This is accomplished by combin- ing Hierarchical Task Network planning, description logics, and a robust execution/monitoring system. Introduction A paradigm shift has been taking place among researchers in various fields of AI, such as perception and manipula- tion in robotics. In recent years, an increasing number of ap- proaches are task-oriented. For example, object grasping is no longer solely dependent on the physical properties of the object to be grasped and those of the manipulator, but now takes into account the purpose of the grasping taking place. For example, grasping an object to pour from it may require a different grasp than transporting it. In the planning field, the task-based perspective predates the current change in other robotics fields. The Hierarchi- cal Task Network (HTN) approach (Erol, Hendler, and Nau 1994a) has enabled researchers to reduce the search space by allowing them to encode the ‘best way’ of carrying out tasks; thereby improving the quality of the plans along the way. Complexity is at a minimum when there are no choices to be made – when there is exactly one way to decompose a non-primitive task into primitive tasks. This may, however, limit the ability to generate a plan: if we have no applica- ble way to decompose a non-primitive task at a given state. We would like to make use of another behavior that humans often exhibit: finding alternative ways to accomplish a task. Quite often, not so much by changing the plan per se (the how), but by making the right substitutions (the with what). For example, by using a mug instead of a glass for drinking or by using a tea kettle instead of a watering can for watering plants. The effect is equivalent to adding a new method to decompose the task that uses the substituted object, and de- pending on the result at execution, annotating it with a pref- erence index. When using lifting, it would enable additional objects as options to ground the methods and operators. In mixed-initiative approaches, this could be seen as a resource assignment (in our case, one that the agent should arrive at autonomously). But, how would we determine which objects to substitute? We propose a modified HTN planning algorithm that reuses the procedural knowledge of the methods and finds object substitutes when necessary and appropriate, thereby mimicking the resourcefulness of human planners and ac- tors. We first answer the question of how the agent rec- ognizes when it should make a substitution. Applying jus- tification structures, borrowed from explanation-based ap- proaches for annotating the derivation process of a plan (Veloso 1994; Fernandez and Veloso 2006) accomplishes this and enables us to understand why a planner made a particular decision and why it may have failed to generate a plan. In cases where planning fails due to a missing ob- ject, the algorithm uses a reasoning process that employs the concept of affordances to expand the domain so that effec- tive alternative choices can be made. Such a choice may be the most appropriate substitution, or the cheapest based on spatial proximity, or some weighted combination of both of these. Affordances describe “opportunities for action” (Gibson 1979). This notion of affordances is retained in this work, although Gibson’s action/perception coupling is not dealt with directly. Gibson’s original definition has been refined by many researchers, but a generally agreed upon interpre- tation narrows the list of action choices to those that an actor is aware of. Using the refined definition, affordances are nei- ther solely a property of the object, nor of the actor, but of their relationship. We adopt Norman’s definition (Norman 2002), and the subsequent extensions of this definition by others, such as (Gaver 1991) and (Hartson 2003)) of per- ceived affordances which allude to “how an object may be interacted with based on the actors’s goals, plans, values, beliefs and past experience” (Norman 2002). We propose to include affordances within the domain model and to repre- sent this in Description Logics (DL) so that we may use the reasoning powers of existing tools to enable the robust and