Human-Robot Team Navigation in Visually Complex Environments John Carff 1 , Matthew Johnson 1 , Eman M. El-Sheikh 2 , and Jerry E. Pratt 1 1 Florida Institute for Human and Machine Cognition 2 Department of Computer Science, University of West Florida Abstract—Current fully autonomous robots are unable to navigate effectively in visually complex environments due to limitations in sensing and cognition. Full teleoperation using current interfaces is difficult and the operator often makes navigation mistakes due to lack of operating environment information and a limited field of view. We present a novel method for combining the sensing and cognition of a robot with that of a human. Our collaborative approach is different from most in that we address bi-directional considerations. It provides the human a mechanism to supplement the robot’s capabilities in a new and unique way and provides novel forms of feedback from the robot to enhance the human’s understanding of the current state of the system and its intentions. Index Terms—Mobile Robots, Navigation, Cognition, Complex Environments I. INTRODUCTION Robots are currently unable to autonomously operate in visually complex environments due to limitations in sensing and cognition. Once outside the structured laboratory environment, the current sensors provide inadequate information for successful autonomous operation. Even the best 3D range finders can miss critical pieces of information and place a large computational burden on the system. However, if a perfect sensor did exist, the lack of cognitive ability to interpret the sensory information would still be a major barrier to navigation. Object recognition is still in its infancy and scene interpretation is still a far off dream. Therefore, to date, the vast majority of robots deployed in urban environments have been fully teleoperated (e.g. Talon and Packbot), placing a large burden on the human operators who are often hindered by poor interfaces [1]. Consistent with much of the recent work being done in this area [2-6], we believe that with a good system design and an effective user interface, a human-robot team navigation system can be faster, more accurate, and more efficient than a purely teleoperated system or a purely autonomous system. In this paper, we describe the design goals for a human-robot team navigation system, methods for achieving those goals, and the implementation of such a system. Section II provides a brief review of other work related to the scope of this project. In Section III, we discuss the design goals and methods for a human-robot team navigation system. In Section IV, we present our implementation of such a system. The paper concludes with a discussion of the uses, benefits, and limitations of our approach, and directions for future work. A. Navigation Challenges for Purely Autonomous Systems Despite the advances made in autonomous navigation, navigating in visually complex environments remains a challenge. In such environments, there are many instances of Turing Test style navigation problems, which seem to require human-level cognition to solve. Figure 1 shows just a few examples of scenes that are simple for humans to interpret, but would thwart today’s robotics systems. On the left is an automatic sliding glass door adjacent to glass windows. To a human familiar with sliding doors, the proper way to enter the building is obvious. This knowledge would be difficult to include in an autonomous system and the shadows and reflections make this a daunting image recognition problem. The image on the right is a fence with a narrow gate that is covered in vines. Discriminating between fixed and moveable obstructions such as the vines is another challenge out of reach of today’s navigation systems. Like typical Artificial Intelligence systems, we could add specific rules for these and other cases but would soon find that there are too many special cases to account for, and that no matter how many special rules we add to our system, we would soon encounter novel situations. Fig. 1. Typical scenes that are trivial for a human to interpret, but challenging for an autonomous system. On the left, an automatic glass sliding door. On the right, a narrow gate with vines. In addition, in a hostile environment, it would be very easy for adversaries to add “Navigation Captchas” to the environment to prevent movement of machines while not