Original research Musculoskeletal-see-through mirror: Computational modeling and algorithm for whole-body muscle activity visualization in real time Akihiko Murai a, * ,1 , Kosuke Kurosaki a , Katsu Yamane b , Yoshihiko Nakamura a a Department of Mechano-Informatics, University of Tokyo; 7-3-1 Hongo, Bunkyo-ku, Tokyo 113-8656, Japan b Disney Research, Pittsburgh/CMU, USA article info Article history: Available online 30 September 2010 Keywords: Musculoskeletal model Real-time visualization Motion capture Electromyography (EMG) Muscle dynamics model abstract In this paper, we present a system that estimates and visualizes muscle tensions in real time using optical motion capture and electromyography (EMG). The system overlays rendered musculoskeletal human model on top of a live video image of the subject. The subject therefore has an impression that he/she sees the muscles with tension information through the cloth and skin. The main technical challenge lies in real-time estimation of muscle tension. Since existing algorithms using mathematical optimization to distribute joint torques to muscle tensions are too slow for our purpose, we develop a new algorithm that computes a reasonable approximation of muscle tensions based on the internal connections between muscles known as neuronal binding. The algorithm can estimate the tensions of 274 muscles in only 16 ms, and the whole visualization system runs at about 15 fps. The developed system is applied to assisting sport training, and the user case studies show its usefulness. Possible applications include interfaces for assisting rehabilitation. Ó 2010 Elsevier Ltd. All rights reserved. 1. Introduction Somatosensory information, including muscle tensions, provides a person with his or her sensation of body posture. And its visualization is important in many fields. Bio-feedback is often used in sports and rehabilitation and improves a health and performance (Patel and North, 1975). Character animations considering this information, e.g. performance captures, appear more realistic and dynamic (Park and Hodgins, 2008; Teran et al., 2005). The real-time estimation and visualization of somatosensory information will strengthen the interactivity, and increase the range of these techniques. A number of algorithms have been proposed for estimating muscle tensions and joint loads using musculoskeletal models (Delp and Loan, 2000; Delp et al., 2007; Rasmussen et al., 2001; Bhargava et al., 2004; Forster et al., 2004; Nakamura et al., 2005). The main problem of muscle tension estimation is that there are an infinite number of solutions to realize a particular joint torque due to the actuation redundancy (there are more actuators than the degrees of freedom (DOF) in the system). It is therefore impossible to obtain precise muscle tension information only from motion data. One of the solutions is to directly measure muscle activations by electromyography (EMG) and use empirical muscle models (Hill, 1938; Stroeve, 1999) to convert the activations to tensions (Nussbaum and Chaffin, 1998; Buchanan et al., 1998; Lloyd and Besier, 2003). In theory, this approach gives the most accurate muscle tension data. It is also very fast because we only need to run the muscle model. In practice, however, it has a number of disad- vantages. First, EMG signals, especially those obtained by nonin- vasive surface EMG measurement, are prone to noise and often mixed up with signals from other muscles. The parameters of the empirical muscle model are not applicable for all the muscles in the human body. Second, the method can only estimate the tensions of muscles with EMG information, which are strictly limited by the number of available EMG channels. It is not realistic either to attach many EMG electrodes to the subject even with the most advanced wireless EMG measurement systems in the market, because the number of EMG channels is limited. Another solution is to use inverse dynamics algorithms devel- oped in robotics to obtain the joint torques of the skeleton and then run numerical optimization to compute the muscle tensions (Rasmussen et al., 2001; Nakamura et al., 2005). Here the joint torques are computed from the motion data and the geometric and inertial parameters of the skeleton model, and the mathematical optimization with physiologically appropriate criteria, e.g. mini- mizing the signal dependent noise (Harris and Wolpert, 1998), distributes them to the muscle tensions. This approach does not * Corresponding author. E-mail addresses: murai@ynl.t.u-tokyo.ac.jp, amurai@disneyresearch.com (A. Murai), kurosaki@ynl.t.u-tokyo.ac.jp (K. Kurosaki), kyamane@disneyresearch.com (K. Yamane), nakamura@ynl.t.u-tokyo.ac.jp (Y. Nakamura). 1 Is now with Disney Research, Pittsburgh. Contents lists available at ScienceDirect Progress in Biophysics and Molecular Biology journal homepage: www.elsevier.com/locate/pbiomolbio 0079-6107/$ e see front matter Ó 2010 Elsevier Ltd. All rights reserved. doi:10.1016/j.pbiomolbio.2010.09.006 Progress in Biophysics and Molecular Biology 103 (2010) 310e317