1 Tactile Object Recognition From Appearance Information Zachary Pezzementi, Student Member, IEEE, Erion Plaku, Caitlin Reyda, Gregory D. Hager, Fellow, IEEE Abstract—This paper explores the connection between sensor- based perception and exploration in the context of haptic ob- ject identification. The proposed approach combines (i) object recognition from tactile appearance with (ii) purposeful haptic exploration of unknown objects to extract appearance infor- mation. The recognition component brings to bear computer vision techniques by viewing tactile sensor readings as images. We present a bag-of-features framework that uses several tactile image descriptors, some adapted from the vision domain, others novel, to estimate a probability distribution over object identity as an unknown object is explored. Haptic exploration is treated as a search problem in a continuous space to take advantage of sampling-based motion planning to explore the unknown object and construct its tactile appearance. Simulation experiments of a robot arm equipped with a haptic sensor at the end-effector provide promising validation, indicating high accuracy in identifying complex shapes from tactile information gathered during exploration. The proposed approach is also validated by using readings from actual tactile sensors to recognize real objects. I. I NTRODUCTION Tactile force sensors, consisting of an array of individual pressure sensors, are becoming common parts of modern manipulation systems. It is generally expected that a new robotic hand design will include tactile force sensors embed- ded in each fingertip and possibly along other surfaces of the hand. The current generation of tactile sensors is also much more capable than previous generations. Resistive sensors are commercially available at resolutions as high as 40x40 per square inch [1], capacitive sensors offer greatly-increased force resolution and repeatability, and recent optical gel sensors [2] offer remarkably high resolutions that depend primarily on the camera being used, size, and other methodological trade-offs between spatial and depth resolution. Given the advancement and ubiquity of tactile force sensors, it becomes important to be able to extract as much information as possible from these sensors about the task at hand. In this work, we use the object recognition task as a benchmark for evaluating the quality of various ways of interpreting tactile force sensor readings. We develop a method to distinguish between objects using only the responses of tactile sensors and compare several representations of tactile information for Zachary Pezzementi and Gregory D. Hager are with the Department of Computer Science and the Laboratory for Computational Sensing and Robotics, Johns Hopkins University, Baltimore, MD 21218. Email: {zap, hager@cs.jhu.edu}. Erion Plaku is with the Department of Electrical Engi- neering and Computer Science, Catholic University of America, Washington, DC 20064. Email: plaku@cua.edu. Caitlin Reyda is with the Department of Mechanical Engineering, Massachussetts Institute of Technology, Cambridge, MA 02139. Email: reyda@mit.edu. (a) Tactile exploration in simulation (b) Tactile images Fig. 1. Depiction of a chess piece being explored by our simulated robotic arm (shown in dark blue) and tactile sensor system (shown in purple). Note that the tactile exploration method does not know the position, orientation, or the geometry of the object. Yellow patches show the sensor placements at which local controllers converged and a local appearance feature was extracted and recorded. The corresponding tactile images are shown to the right. this purpose. The effectiveness of the method is demonstrated by recognizing a set of complex 3D objects in simulation and a set of raised letters both in simulation and using real sensors. Our general approach is to interpret tactile sensor readings as “tactile images”, which measure a patch of the surface of an object. In previous work, we characterized a set of tactile sensors from Pressure Profile Systems [3] and developed a simulator to emulate that class of tactile sensors’ response in interactions with rigid objects [4]. Tactile sensors were found to be modeled well as camera systems that detected depth information, modified by a point spread function dependent on the thickness of a covering material. Now we use the same sensor model, but expand the simulation to include the full robotic exploration task, with tactile sensing as the sole form of feedback, as illustrated in Fig. 1. By thinking of sensor readings as images, we bring to bear a large body of work from computer vision. The interpretation of the information in these force images is somewhat simpler than in the visual case, since there are no perspective effects and there is only one channel of intensity information. The collection of images, however, is considerably more difficult, since each small patch must be obtained by actively interacting with the environment, while hundreds of features can be extracted from a single passive image in the visual case. In order to get useful tactile force readings, we draw from recent advances in sampling-based motion planning [5]–