TRAILBLAZING: VIDEO PLAYBACK CONTROL BY DIRECT OBJECT MANIPULATION Don Kimber 1 , Tony Dunnigan 1 , Andreas Girgensohn 1 , Frank Shipman 2 , Thea Turner 1 , Tao Yang 1,3 1 FX Palo Alto Laboratory, Palo Alto, CA, 94304, USA 2 Texas A&M University, College Station, TX 77843-3112, USA 3 Northwestern Polytechnical University, Xi’an, 710072, China lastname@fxpal.com shipman@csdl.tamu.edu yangtaonwpu@163.com ABSTRACT We describe a new interaction technique that allows users to control nonlinear video playback by directly manipulating objects seen in the video. This interaction technique is simi- lar to video “scrubbing” where the user adjusts the playback time by moving the mouse along a slider. Our approach is superior to variable-scale scrubbing in that the user can con- centrate on interesting objects and does not have to guess how long the objects will stay in view. Our method relies on a video tracking system that tracks objects in fixed cameras, maps them into 3D space, and handles hand-offs between cameras. In addition to dragging objects visible in video windows, users may also drag iconic object representations on a floor plan. In that case, the best video views are se- lected for the dragged objects. 1. INTRODUCTION An important affordance of digital video is that it supports nonlinear viewing in whatever manner is most suitable to a given task. Particularly for purposes such as process analy- sis, sports analysis or forensic surveillance tasks, some por- tions of the video may be skimmed over quickly while other portions are viewed repeatedly many times, at various speeds, playing forward and backward in time. Scrubbing, a method of controlling the video frame time by mouse mo- tion along a time line or slider, is often used for this fine level control, allowing a user to carefully position the video at a point where objects or people in the video are in certain positions of interest or moving in a particular way. Scrubbing and off-speed playback, such as slow motion or fast forward, are useful but have limitations. In particular, no one playback speed, or scale factor for mapping mouse motion to time changes, is appropriate for all tasks or all portions of video. Adding play speed controls and the ability to zoom in on all or some of the timeline helps, and some authors describe variable scale scrubbing where mouse mo- tion orthogonal to the slider sets the scale [5]. But these features can be confusing and distracting. Instead of directly controlling what they view, users spend time focusing on control features of the interface. Some researchers have ad- dressed this using variable speed playback with speed de- termined by the amount of motion or new information at each time of the video [7]. These schemes essentially re- index the video from time t to some function s(t), where for example s(t) is the cumulative information by time t, or the distance a tracked object moved. We have experimented with such schemes for variable scale scrubbing, where in addition to a time slider, the user is provided with another slider for s. Since there are various measures of change, or various objects to track, multiple sliders can be provided. However, rather than indirect scrubbing control by mul- tiple sliders, a more natural interface is based on the reach- through-the-screen metaphor proposed in [2] — simply let users directly grab and move objects along their trails. This may be done directly in a video window or in a floor plan view which schematically indicates positions of people by icons (see Fig. 1). For example, a user reviewing surveil- Fig. 1. Viewer showing trails in video and floor plan view. The user may scrub video by dragging a person along their path in the video window or on the floor plan.