Submitted to International Joint Conference on Neural Networks, 2002. A Recurrent Neural Network Approach to Virtual Environment Latency Reduction Aaron Garrett Knowledge Systems Laboratory Jacksonville State University Jacksonville, AL 36265 aarong@ksl.jsu.edu Mario Aguilar, Ph. D. Knowledge Systems Laboratory Jacksonville State University Jacksonville, AL 36265 marioa@ksl.jsu.edu Yair Barniv, Ph. D. NASA/Ames Research Center Human Factors Research Division Moffet Field, CA 94035-1000 ybarniv@mail.arc.nasa.gov Abstract – We present a recurrent neural network system designed to predict future angular acceleration of the human head from current angular acceleration data. These predictions can be used to supplement head tracking in virtual environments in order to reduce latency and increase tracking accuracy, thus enhancing the user’s performance and comfort. I. INTRODUCTION With increasingly smaller and faster computing machinery, the use of virtual environments (VE) in everyday applications is becoming more a matter of fact, rather than a matter of science fiction. However, using virtual environments is not without its difficulties. One of the most notable problems facing current VE applications is the perceptible latency that is experienced by the user as the computer displays are updated. Such perceptible latency has been shown to have undesirable effects on users of virtual environments, including a lack of accuracy during tracking tasks, motion sickness, and loss of immersion [1]. Ellis, et al, have shown that visual latency, more than spatial sensor distortion or low update rates, is the primary cause of RMS tracking errors [2]. This latency comes in three different forms – rendering lag, application-dependent processing lag, and user input device lag [3]. Rendering lag is due to the amount of time it takes for the computer screen to be updated. Application-dependent processing lag is the latency that arises from the computation of the three-dimensional model. This latency is entirely dependent on the complexity of the model. Finally, user input device lag is the lag introduced from the necessary communication between the position and orientation tracking system and the VE application. Because of its relevance to our current work, in this paper we will only consider this final type of lag. User input device lag can range between 10ms and 120 ms [3]. Because of the detrimental effects that often result from lag in virtual environments, many motion trackers implement a prediction mechanism within the tracker itself that attempts to compensate for the device lag. This prediction mechanism is often no more than a crude averaging and extrapolation from the current movement to the next movement in time. While such a scheme can be useful and effective in certain situations, it manifests serious problems if the user moves quickly or changes velocity. Thus, in order to avoid experiencing the cyber-sickness caused by lag, the user must make slow, deliberate movements [4]. Also, using these approaches, predictions out to a useful range (i.e., 20–40 ms) are subject to very large errors. If differential acceleration data (as generated by linear accelerometers or rate gyros) is used, more accurate extrapolation systems can be created because of the higher resolution of the original data. We will discuss these extrapolation methods in more detail in Section III. In an effort to improve upon the prediction within the tracker, itself, other prediction mechanisms have been implemented which make use of Kalman filters. For instance, Friedmann, et al, used Kalman filters to predict the position, velocity, and rotations of a Polhemus sensor [4]. Their results show that accurate predictions can be made except at points where the user changes acceleration and/or direction very sharply. Such unexpected changes were also the major cause of problems for the on-board prediction method of the trackers themselves. Friedmann, et al, dealt with this problem by using a multiple model approach. In this approach, several different Kalman filters were created to accommodate different types of user movements. At each instance of prediction, the filter with the maximum likelihood was used. While the above study dealt with both positional prediction and rotational prediction, it has been shown that users are much more affected by rotational latency than positional latency. This has been attributed to two main causes. First, because of the limitations of most motion trackers, positional changes are limited to a small range. Thus, users tend to make such changes infrequently. Second, rotational changes generally cause more noticeable changes in the scene than do positional changes [5]. Because of this fact, it seems reasonable to focus current motion prediction research to deal with only the rotational accelerations of the user. In fact, Ellis, et al, carried out an experiment in which Kalman filters were used to predict only the angular accelerations of user head motions within a virtual environment [6]. These angular accelerations are based on approximations using the Polhemus tracker measurements, not on actual rate gyro data which would be much more accurate. Using differential acceleration data collected from rate gyros is indeed an extremely powerful approach to head movement prediction. However, to the best of our knowledge, linear approximation techniques such as Kalman filters have only been applied to approximated acceleration data. In an effort to make use of collected differential acceleration data and to move beyond the linear filtering techniques that have been applied to date, we present a recurrent neural network approach to the problem of angular acceleration prediction. However, rather than using angular acceleration about all three axes, we constrain the problem by using angular acceleration about the z-axis only. (This would