Dual Arm Estimation for Coordinated Bimanual Manipulation Paul Hebert, Nicolas Hudson, Jeremy Ma Jet Propulsion Laboratory, California Institute of Technology Pasadena, CA, 91109, USA Joel W. Burdick California Institute of Technology Pasadena, CA, 91125, USA Abstract— This paper develops an estimation framework for sensor-guided dual-arm manipulation of a rigid object. Using an unscented Kalman Filter (UKF), the approach combines both visual and kinesthetic information to track both the manipulators and object. From visual updates of the object and manipulators, and tactile updates, the method estimates both the robot’s internal state and the object’s pose. Nonlinear constraints are incorporated into the framework to deal with the an additional arm and ensure the state is consistent. Two frameworks are compared in which the first framework run two single arm filters in parallel and the second consists of the augment dual arm filter with nonlinear constraints. Experiments on a wheel changing task are demonstrated using the DARPA ARM-S system, consisting of dual Barrett TM WAM manipulators. I. I NTRODUCTION Large, heavy, and cumbersome objects often requires two or more manipulators for dextrous manipulation. With such objects, single arm grasps are often instable or lack enough force to maintain a proper grasp. In addition, in unstructured environments, sensor-based estimation is a key requirement for autonomous robots. Estimation of both the robot’s internal state and the object to be manipulated is necessary for both planning and closed-look control in accurate manipulation. As lower cost manipulators [1] become more prevalent rather than expensive, rigid and highly geared manipulators, heavier object will induce larger deviations of the end effec- tor. As a result, visual manipulator tracking is a necessity. Moreover, for tasks that involve precise manipulation, kines- thetic information is especially needed to close the visual loop between the object the manipulator. This paper presents a novel estimation framework for dual-arm object manipulation by visually tracking both the object and manipulator arms and connecting the two through kinesthetic updates. Our approach uses an unscented Kalman Filter (UKF) to fuse multiple visual sensor measurements and tactile information, if they are available. As a result of the ad- ditional manipulator, nonlinear constraints are incorporated into the filtering framework to constrain the state imposed by the additional contact information from the two arms. As humanoid robot’s become more prevalent and working in unstructured environments, dual-arm object estimation is a critical component to safe and successul operation. II. RELATED WORK Visual tracking and pose estimation for sensor-based robotic manipulation has been studied before. Most work has Fig. 1: DARPA ARM-S dual arm manipulation robot with wheel grasped in hands. focused on visual based estimation [2]–[5] for manipulation or single sensor estimation such as tactile [6]–[9]. Other work studied the combination and fusion of multiple sensors for manipulation, such as vision, tactile and/or force [10]–[14]. There has been work with dual-arm manipulation but most dealt with planning and control [15]–[18]. In addition, most work regarding estimation in dexterous manipulation has focused only on single arm robots [19]–[22]. Coordinated manipulation is not limited to anthropomorphic robot arms, but may consist of bimanual industrial manipulators [23], or even multiple robot fingers [24]. For a survey of dual arm manipulation, the reader may consult Kragic et al [25]. To the authors’ knowledge, an estimation framework for dual-arm object manipulation has not been addressed previ- ously. III. APPROACH The estimation approach is based on the following as- sumptions and assumed general system configuration (see Figure 2). A serial chain manipulator is rigidly affixed to a base, or to a (possibly moving) torso. We assume that the manipulator possesses 6 or more internal degrees of freedom (though the technique could be adapted to a kinematically insufficient manipulator which possesses 5 or fewer DOF.).