Predicting Grasp Success in the Real World - A Study of Quality Metrics and Human Assessment Carlos Rubert a,∗ , Daniel Kappler b , Jeannette Bohg c , Antonio Morales a a Robotic Intelligence Laboratory at the Department of Computer Science and Engineering, Universitat Jaume I of Castellon, Spain b Google X Robotics, Mountain View, CA, USA. ** c Department of Computer Science, Stanford University, USA Abstract Grasp quality metrics aim at quantifying different aspects of a grasp configuration between a specific robot hand and object. They produce a numerical value that allows to rank grasp configurations and optimize based on them. Grasp quality metrics are a key part of most analytical grasp-planning approaches. Additionally, they are often used to generate ground-truth labels for synthetically generated grasp exemplars required for learning-based approaches. Recent studies have highlighted the limitations of grasp quality metrics when used to predict the outcome of a grasp execution on a real robot. In this paper, we systematically study how well seven commonly-used grasp quality metrics perform in the real world. To this end, we generated two datasets of grasp candidates in simulation, each one for a different robotic system. The quality of these synthetic grasp candidates is quantified by the aforementioned metrics. For validation, we developed an experimental procedure to accurately replicate grasp candidates on two real robotic systems and to evaluate the performance of each grasp. Given the resulting datasets, we trained different classifiers to predict grasp success using only grasp quality metrics as input. Our results show that combinations of quality metrics can achieve up to a 85% classification accuracy for real grasps. Keywords: Grasping, Grasp simulation, Machine learning, Prediction model, Real grasp execution. 1. Introduction Grasp quality metrics are computational tools that eval- uate grasp configurations consisting of contact points be- tween the robot end-effector and the object surface. These metrics quantify grasp quality based on the measured forces and torques exerted at the contact points. Desirable prop- erties of a grasp that are evaluated by existing quality metrics are force-closure, equilibrium, dexterity, stability and others [43, 37]. Grasp quality metrics aspire to predict the outcome of a grasp on a specific object when executed with a real robotic system. They are central to analytic approaches in grasp planning, which are formulated as an optimization problem over grasp configurations given the object and robot hand dynamic models [5]. Another common use of grasp quality metrics is to generate the ground- truth la- bels for synthetically generated grasp configurations. The resulting data sets are then used for learning-based ap- * Corresponding Author ** The majority of this work has been conducted while the author was with the Autonomous Motion Department at the MPI for Intel- ligent Systems, T¨ ubingen, Germany. Email addresses: carlos.rubert@uji.es (Carlos Rubert), daniel.kappler@gmail.com (Daniel Kappler), bohg@cs.stanford.edu (Jeannette Bohg), morales@uji.es (Antonio Morales) proaches in grasp planning e.g. [18, 27, 28]. Unlike ana- lytic approaches, learned grasp planning models often take partial sensory data as input instead of full 3D object mod- els. They are also able to generalize over objects that the trained model has not yet seen. Multiple studies have emphasized the limitations of classic grasp quality metrics when predicting grasp suc- cess in the real world [2, 44, 9, 18]. First, quality metrics rely on precise models of robot hands and objects. These are not always available, in particular for the wide variety of objects that exist in the real world. Second, the desired contacts have to be precisely achieved for the grasp quality metric to be valid. This is challenging due to the inherent inaccuracy of robot control, noise in sensor measurements and other sources of uncertainty. And third, individual quality metrics typically focus only on specific aspects of the physical interaction. Real executions are however af- fected by a variety of aspects that may not be taken into account by a particular metric. Thus, grasp configurations may fail in the real world despite a high-quality value. For example, most commonly used quality metrics consider only the moment after which contact is established be- tween robot hand and object. However, when establishing the grasp and lifting the object other aspects will have a large influence on the success of the grasp. The main contribution of this work is to evaluate to what extent grasp metrics obtained in simulation transfer Preprint submitted to Elsevier September 10, 2019