Interactive Gaze Driven Animation of the Eye Region * Debanga R. Neog Jo˜ ao L. Cardoso Anurag Ranjan Dinesh K. Pai Sensorimotor Systems Laboratory, Department of Computer Science, University of British Columbia d (a) (b) (c) (d) Figure 1: We describe a complete pipeline to model skin deformation around the eyes as a function of gaze and expression parameters. The face is rendered with our WebGL application in real-time and runs in most modern browsers (a) and mobile devices (b). An example of automatic facial animation generation while watching a hockey video is shown (c-d). The character starts watching the match with a neutral expression (c) and gets concerned when a goal is scored (d). Eye movements were generated automatically using salient points computed from the hockey video. Abstract We propose a system for real-time animation of eyes that can be interactively controlled in a WebGL enabled device using a small number of animation parameters, including gaze. These animation parameters can be obtained using traditional keyframed animation curves, measured from an actor’s performance using off-the-shelf eye tracking methods, or estimated from the scene observed by the character, using behavioral models of human vision. We present a model of eye movement, that includes not only movement of the globes, but also of the eyelids and other soft tissues in the eye re- gion. The model includes formation of expression wrinkles in soft tissues. To our knowledge this is the first system for real-time ani- mation of soft tissue movement around the eyes based on gaze in- put. CR Categories: I.3.7 [Computer Graphics]: Three-Dimensional Graphics and Realism—Animation I.3.8 [Computer Graphics]: Three-Dimensional Graphics and Realism—Applications Keywords: WebGL, human animation, eyes, soft tissue, gaze. 1 Introduction Whether or not the Bard actually said “The eyes are the window to your soul,” the importance of eyes is well recognized, even by the general public. Alfred Yarbus’s influential work in the 1950s and * http://www.cs.ubc.ca/research/eyemoveweb3d16/ Permission to make digital or hard copies of all or part of this work for per- sonal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstract- ing with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from permissions@acm.org. c  2016 Copyright held by the owner/author(s). Publication rights licensed to ACM. Web3D ’16,, July 22 - 24, 2016, Anaheim , CA, USA ISBN: 978-1-4503-4428-9/16/07 $15.00 DOI: http://dx.doi.org/10.1145/2945292.2945298 1960s quantified this intuition; using an eye tracker, he noted that observers spend a surprisingly large fraction of time fixated on the eyes in a picture. The eyes of others are important to humans be- cause they convey subtle information about a person’s mental state (e.g., attention, intention, emotion) and physical state (e.g., age, health, fatigue). Consequently, eyes are carefully scrutinized by humans and other social animals. Creating realistic computer gen- erated animations of eyes is, therefore, a very important problem in computer graphics. But what is it about eyes that conveys this important information to observers? To discuss this, we need to introduce some terms. The colloquial term “eye” is not sufficiently precise. It includes the globe, the approximately spherical optical apparatus of the eye that includes the colorful iris. The globe sits in a bony socket in the skull called the orbit. The term “eye” also usually includes the upper and lower eyelids and periorbital soft tissues that surround the orbit, including the margins of the eyebrows. When we refer to the eye we mean all of these tissues, and we will use the more specific term where appropriate. The most obvious property of eyes is gaze, that is, what the eyes are looking at. This is entirely determined by the position and ori- entation of each globe relative to the scene. Almost all previous work in animation of eyes has been on animating gaze, with some recent attention paid to the appearance of the globe and iris, and the kinematics of blinks (see Sec. 2 for a review of the related work). For instance, an excellent recent survey of eye modeling in anima- tion [Ruhland et al. 2014] does not even mention the soft tissues or wrinkles surrounding the eyes. Gaze is clearly important, but the soft tissues of the eye also con- vey a lot of information. For instance, the well known “Reading the Mind in the Eyes,” test developed by Autism researcher Baron- Cohen [2001] used still images of people’s eyes, without any con- text about what they were looking at, and hence little gaze infor- mation 1 . Yet, normal observers are able to read emotion and other attributes from these images alone. We hypothesize that the state of soft tissues surrounding the eyes is a major source of informa- 1 You can try it yourself at http://well.blogs.nytimes.com/2013/10/03/well- quiz-the-mind-behind-the-eyes/