Adaptive 2.5D Visual Servoing of Kinematically Redundant Robot Manipulators ∗ Y. Fang † , A. Behal † , W. E. Dixon ‡ , and D. M. Dawson † † Department of Electrical & Computer Engineering, Clemson University, Clemson, SC 29634-0915 ‡ Eng. Science and Tech. Div. - Robotics, Oak Ridge National Lab., P.O. Box 2008, Oak Ridge, TN 37831-6305 email: yfang, abehal, ddawson@ces.clemson.edu; dixonwe@ornl.gov Abstract In this paper, the 3-Dimensional (3D) position and orientation of a camera held by the end-effector of a robot manipulator is reg- ulated to a constant desired position and orientation despite (i) the lack of depth information of the actual or desired camera position from a target, (ii) the lack of a 3D model of the target object, and (iii) parametric uncertainty in the dynamic model of the robot ma- nipulator. Specifically, by fusing 2D image-space and 3D task-space information (i.e., 2.5D visual servoing) while actively adapting for unknown depth information, a task-space kinematic controller is de- veloped that is proven to ensure asymptotic regulation of the posi- tion and orientation of the camera. Based on the desire to enhance the robustness of the control design, the integrator backstepping approach is then utilized to develop a joint torque control input to ensure asymptotic regulation of the position and orientation of the camera, which is held by the end-effector of a kinematically re- dundant robot manipulator, despite parametric uncertainty in the dynamic model of the robot. The stability of each controller is proven through a Lyapunov-based stability analysis. 1 Introduction Motivated by the desire to enable robotic systems with a greater sense of perception and ability to operate in unstructured environ- ments, researchers have actively investigated the use of visual servo- ing control systems (i.e., using information obtained from a camera system to provide position and orientation information about the robot and it’s environment for use in a control scheme). The results from this research can be broadly divided into Image-Based Visual Servoing (IBVS) and Position-Based Visual Servoing (PBVS) tech- niques. In PBVS, features are extracted from the camera image and then related to the task-space through the calibrated image Jacobian. The resulting task-space error system is then utilized by the control system. Since the control is calculated based on the task-space error system, inaccuracies in camera calibration will lead to inaccuracies in the 3-Dimensional (3D) task-space recon- struction and ultimately in the task execution. Moreover, since the image-space information is not utilized by the controller, the im- age features may exit the camera’s field-of-view resulting in a loss of stability. In contrast to PBVS, IBVS schemes define an image- space error system that is utilized by the controller. Based on the fact that IBVS controllers servo from the image-space error, this approach ensures that the image will remain in the camera field- of-view and there is conjecture that this approach facilitates some measure of robustness to calibration errors; however, IBVS tech- niques have problems related to singularities in the image Jacobian, and local minimas may be reached rather than the actual desired position and orientation. In addition to the previous shortcomings, a common characteristic of many PBVS and IBVS techniques is that an accurate 3D model of the environment (or target image) is often required (see [1] for a more in-depth discussion regarding PBVS and IBVS). Several researchers have recently developed partitioned ap- proaches that exploit a combination of 3D task-space information and 2D image-space information to overcome many of the short- comings of traditional PBVS and IBVS approaches. For example, in the series of papers by Malis and Chaumette (e.g., [2, 3, 15, 16]) ∗ This research was supported in part by the Eugene P. Wigner Fellowship Program of the Oak Ridge National Laboratory (ORNL), managed by UT-Battelle, LLC, for the U.S. Department of Energy (DOE) under contract DE-AC05-00OR22725 and in part by the U.S. DOE Environmental Management Sciences Program (EMSP) projects ID No. 82794 and ID No. 82797 at ORNL, by ONR Project No. N00014-00-F-0485 at ORNL, and by U.S. NSF Grant DMI-9457967, ONR Grant N00014-99-1-0589, a DOC Grant, and an ARO Automotive Center Grant. various kinematic control strategies (coined 2.5D visual servo con- trollers) exploit the fact that the interaction between translation and rotation components can be decoupled through a homography. Specifically, information from the 3D task-space (obtained either through a given 3D model or more interestingly through a projec- tive Euclidean reconstruction) is utilized to regulate the rotation error system while information from the 2D image-space is utilized to control the translation error system. This control approach in- corporates the advantages of both PBVS and IBVS; however, many of the disadvantages of the traditional approaches are avoided [16]: (i) an accurate 3D model of the environment (or target image) is not required, (ii) the image is guaranteed to remain in the camera field-of-view, (iii) local minima can be avoided, and (iv) singularities only exist in the image Jacobian in degenerate cases. In [8], Deguchi describes how an interaction between the translation and rotation of images can result in slower transient performance due to ineffi- cient camera motions used to reach the desired image. Based on this observation, Deguchi then proposes two algorithms to decouple the rotation and translation components using a homography and an epipolar condition. Specifically, Deguchi decomposes the trans- lation and rotation components through a homography and states that the 2.5D controller given in [3] can be utilized, and as an alter- nate method, Deguchi develops a kinematic controller that utilizes task-space information to regulate the translation error and image- space information to regulate the rotation error. More recently, Corke and Hutchinson [7] developed a new hybrid image-based vi- sual servoing scheme that decouples rotation and translation com- ponents about the z-axis from the remaining degrees of freedom to address the so called “Chaumette Conundrum,” in which desirable image-space trajectories result in undesirable Cartesian trajectories. One drawback of the controllers given in [2, 3, 7, 8, 14, 15, 16] is that each of the results claim that a constant estimate of the aforemen- tioned depth information can be utilized in lieu of the exact value (although, no stability analysis is provided to support this claim). That is, as stated in [16], an off-line learning stage is required to estimate the distance of the desired camera position to the refer- ence plane. Motivated by the desire to compensate for the afore- mentioned depth information, [5] developed an adaptive kinematic controller to ensure uniformly ultimately bounded (UUB) set-point regulation of the image point errors while compensating for the un- known depth information, provided conditions on the translational velocity and the bounds on uncertain depth parameters are satis- fied. In [19], Taylor et al. developed a kinematic controller that uti- lizes a constant, best-guess estimate of the calibration parameters to achieve local set-point regulation; although, several conditions on the rotation and calibration matrix are required. Most control approaches do not account for the inevitable mis- match between the actual and desired camera translation and rota- tion velocity caused by the nonlinear kinematics and dynamics of the robot manipulator, and hence, reduce the problem to that of kinematic control that simply reacts to image-space errors (e.g., all of the aforementioned research efforts have taken this approach). In [6], Corke and Good presented one of the first results to highlight the advantages of incorporating the robot dynamics in the over- all control design. Motivated by the results in [6], several other researchers have proposed vision-based controllers that incorporate the dynamics of the robot. Most of this research has targeted vision- based robotic systems in which the robot is constrained to move in a plane such that the optical axis of the camera remains perpendic- ular to the robot workspace (e.g., see [9, 12, 20, 21]). Some of the few control designs that take the robot dynamics into account for the 3D visual servoing problem are given in [4, 13]. In this paper, we relate feature points extracted from images taken from the desired and current camera position and orientation through a homography. In a similar manner as in [2, 3, 8, 14, 15, 16], we then decompose the homography into translation and rotation components. Based on this homography decomposition, we then develop a task-space kinematic controller that is inspired by [16]. Specifically, in a similar manner as in [16], the kinematic controller utilizes projected 3D task-space information to regulate the rota- tion error system and 2D image-space information to regulate the translation error system. Unlike the controller given in [16], the Proceedings of the 41st IEEE Conference on Decision and Control Las Vegas, Nevada USA, December 2002 ThM03-2 0-7803-7516-5/02/$17.00 ©2002 IEEE 2860