Vision-based Follow-the-Leader Noah Cowan † Omid Shakernia ‡ Ren´ e Vidal ‡ Shankar Sastry ‡ † Department of Mechanical Engineering Johns Hopkins University Baltimore, MD 21218 USA ‡ Department of Electrical Engineering and Computer Science University of California Berkeley, CA 94720 USA Abstract—We consider the problem of having a group of nonholonomic mobile robots equipped with omnidirectional cameras maintain a desired leader-follower formation. Our approach is to translate the formation control problem from the configuration space into a separate visual servoing task for each follower. We derive the equations of motion of the leader in the image plane of the follower and propose two control schemes for the follower. The first one is based on feedback linearization and is either string stable or leader- to-formation stable, depending on the sensing capabilities of the followers. The second one assumes a kinematic model for the evolution of the leader velocities and combines a Luenberger observer with a linear control law that is locally stable. We present simulation results evaluating our vision- based follow-the-leader control strategies. I. I NTRODUCTION Birds flock and fish school without explicit communi- cation between individuals. Vision seems to be a critical component in animals’ abilities to respond their neighbors’ motions so that the entire group maintains a coherent formation. Our long-term goal involves enabling groups of mobile robots to visually maintain formations in the ab- sence of communication, as depicted in Figure 1. Towards that end, we propose and compare two new vision-based controllers that enable one mobile robot to track another, which we dub vision-based follow-the-leader. Thanks to recent advances in computer vision, one can now address formation control without using explicit communication. For example, Vidal et al. [23] consider a formation control scenario in which motion segmentation techniques enable each follower to estimate the image- plane position and velocity of the other robots in the formation, subsequently used for omnidirectional image- based visual servoing (for a tutorial on visual servoing, see [12]). However, the control law in [23] suffers from singular configurations due to nonholonomic constraints on the robots’ kinematics. This paper compares two new visual servo controllers that employ a modification of the image-based coordinate system first presented in [2], which we have modified for omnidirectional imaging. Our first controller builds directly on the work of Desai et al. [6] and Das et al. [4], who use input-output feedback linearization on a clever choice of output function – the so-called “separation and bearing” – defined in terms of the absolute Cartesian configurations of the follower and its leader. We show that their approach, which avoids the above-mentioned singularity [23], can be implemented in image-coordinates, thus simplifying the state estimation problem and enabling the use of image-based visual servoing. Our controller inherits asymptotic convergence from [4] when the forward speed of the leader is known, and, additionally, is leader-to-formation stable (LFS) even without an estimate of the leader velocity. Our second controller uses a linearization of the leader- follower dynamics, in image-plane coordinates. We note that as long as the leader keeps moving (as also required by [4]), one can stabilize the leader-follower formation with a simple linear control scheme. Of course, the linear controller affords only local convergence guarantees, but our simulations suggest that the controller is quite robust and has a large domain of attraction. Its relative simplicity makes it, in some ways, an appealing alternative to the nonlinear control scheme also presented. Both controllers are exponentially stable and hence we are able to take advantage of results in string and input-to-state stability of formations, which we review in Section I-B. Mirror CCD Fig. 1. An omnidirectional vision-based formation of mobile robots. A. Organization In Section II we review the imaging model of a central panoramic camera. In Section III we derive the equations of motion of the leader in the image plane of the follower, and modify the coordinate system of [2] for the present context. In Section IV we design a feedback control