2184 IEEE TRANSACTIONS ON BIOMEDICAL ENGINEERING, VOL. 60, NO. 8, AUGUST 2013 Markerless Motion Capture and Measurement of Hand Kinematics: Validation and Application to Home-Based Upper Limb Rehabilitation Cheryl D. Metcalf ∗ , Member, IEEE, Rebecca Robinson, Adam J. Malpass, Tristan P. Bogle, Thomas A. Dell, Chris Harris, and Sara H. Demain Abstract—Dynamic movements of the hand, fingers, and thumb are difficult to measure due to the versatility and complexity of movement inherent in function. An innovative approach to mea- suring hand kinematics is proposed and validated. The proposed system utilizes the Microsoft Kinect and goes beyond gesture recog- nition to develop a validated measurement technique of finger kine- matics. The proposed system adopted landmark definition (vali- dated through ground truth estimation against assessors) and grip classification algorithms, including kinematic definitions (validated against a laboratory-based motion capture system). The results of the validation show 78% accuracy when identifying specific markerless landmarks. In addition, comparative data with a pre- viously validated kinematic measurement technique show accuracy of MCP ± 10 ◦ (average absolute error (AAE) = 2.4 ◦ ), PIP ± 12 ◦ (AAE = 4.8 ◦ ), and DIP ± 11 ◦ (AAE = 4.8 ◦ ). These results are notably better than clinically based alternative manual measure- ment techniques. The ability to measure hand movements, and therefore functional dexterity, without interfering with underlying composite movements, is the paramount objective to any bespoke measurement system. The proposed system is the first validated markerless measurement system using the Microsoft Kinect that is capable of measuring finger joint kinematics. It is suitable for home-based motion capture for the hand and, therefore, achieves this objective. Manuscript received November 19, 2012; revised January 25, 2013; accepted February 25, 2013. Date of publication March 7, 2013; date of current version July 13, 2013. This work was supported by the University of Southampton. Asterisk indicates corresponding author. ∗ C. D. Metcalf is with the Faculty of Health Sciences, University of Southampton, Southampton, SO17 1BJ, U.K. (e-mail: c.d.metcalf@ soton.ac.uk). R. Robinson was with the Department of Electronics and Computer Sci- ences, University of Southampton, Southampton, SO17 1BJ, U.K. She is now with Roke Manor Research Ltd., Romsey, SO51 0ZN, U.K. (e-mail: rebecca.robinson@roke.co.uk). A. J. Malpass was with the Department of Electronics and Computer Sci- ences, University of Southampton, Southampton, SO17 1BJ, U.K. He is now with Dialog Semiconductor, Swindon, SN5 7XB, U.K. (e-mail: adammalpass@ hotmail.co.uk). T. P. Bogle was with the Department of Electronics and Computer Sciences, University of Southampton, Southampton, SO17 1BJ, U.K. He is now with WheelRight, Oxford, OX5 1PF, U.K. (e-mail: tristanb1989@gmail.com). T. A. Dell was with the Department of Electronics and Computer Sci- ences, University of Southampton, Southampton, SO17 1BJ, U.K. He is now with McLaren Electronic Systems Ltd., Woking, GU21 4YH, U.K. (e-mail: t.dell@me.com). C. Harris is with the Roke Manor Research Ltd., Romsey, SO51 0ZN, U.K. (e-mail: chris.harris@roke.co.uk). S. H. Demain is with the Faculty of Health Sciences, University of Southamp- ton, Southampton, SO17 1BJ, U.K. (e-mail: shd@soton.ac.uk). Color versions of one or more of the figures in this paper are available online at http://ieeexplore.ieee.org. Digital Object Identifier 10.1109/TBME.2013.2250286 Index Terms—Hand kinematics, markerless, Microsoft Kinect, telerehabilitation. I. INTRODUCTION H AND movements and hand function are intrinsic to qual- ity of life. Dexterous ability is fundamental to gesture, communication, independence, and manipulation of, and in- teraction with, objects and the environment. When functional ability is impaired, the prescription of rehabilitation exercises at home is a solution that can increase the intensity of practice and optimize recovery potential. The repetitive nature of home-based rehabilitation programs, such as those for stroke patient care, can be monotonous and often difficult for individuals to complete. Assessment of prac- tice is essential for monitoring recovery, adapting prescriptions based on improvement or decline, and therefore providing more patient-centered care. Home-based rehabilitation and telecare are advocated by many government health organizations [1]–[3] to increase the throughput of patients for oversubscribed inter- national healthcare systems. This, however, is yet to be realized in mainstream healthcare. One important limiting factor is the current inability to mon- itor and measure hand movements. Systems capable of mea- suring the fine dexterity of the hand are, therefore, required. Rehabilitation professionals require detailed data to monitor clinical progress and modify treatments, while patients need real-time feedback to correct their movements and to stimu- late motivation. Any telecare system must, therefore, provide enough feedback to replace verbal feedback usually provided by therapists. Motion capture systems have been used in a wide range of industries, including medical, occupational, sports, and enter- tainment [4]–[6]. Four motion capture systems are currently used for hand capture: instrumented gloves [7], wrist-worn laser systems [8], inertial systems [7], and traditional opti- cal systems [9], [10]. Glove-based systems provide a simple, quick measurement of hand position and do not suffer from occlusions. However, wearing and removing gloves can be dif- ficult or impossible for patients with hand deformities, spastic- ity, and contractures. In contrast, inertial systems, such as the Nintendo Wii, are widely used in rehabilitation centers [11]. However, these allow “trick” movements, i.e., flicking the wrist to substitute whole-arm movements and, therefore, have limited application for unsupervised home-based rehabilitation [12]. 0018-9294/$31.00 © 2013 IEEE