Proceedings of the 12 th International Conference on Auditory Display, London, UK June 20 - 23, 2006 SONICALLY-ENHANCED MOUSE GESTURES IN THE FIREFOX BROWSER Louise Midgley & Paul Vickers Northumbria University School of Computing, Engineering, & Information Sciences Pandon Building, Camden St., Newcastle upon Tyne, NE2 1XE, UK paul.vickers@unn.ac.uk ABSTRACT The use of the mouse to allow interaction via gestures has attracted much interest recently and the popular FIREFOX web browser has been enhanced by an extension supporting mouse gestures. These gestures reveal an interaction problem: feedback is limited (often only a terse message in the browser’s status bar) and navigation errors easily result when the user unknowingly executes a gesture when trying to accomplish some other task (e.g. copying text from a web page). This paper describes an attempt to improve the inter- action experience by adding auditory cues to inform the user about the progress and progression of gestural commands. FIREFOX was chosen as it has an open extension architecture that is easily mod- ified. Preliminary trials indicate increased user satisfaction and comprehension when using auditory-enhanced gestures over the non-enhanced gestures. 1. GESTURES A gesture is “a mark or stroke that causes a command to execute” [1]. Gestures are common in mobile interfaces ([2, 3]) as they offer a usable alternative to the keyboard and mouse, though a high recognition accuracy is needed to prevent rejection by users [2, 1]. The most common gestures are driven by a stylus, finger or mouse [1]. Moyle & Cockburn [4] examined mouse gestures for web navigation. They were motivated by usability issues associ- ated with the “back” and “forward” command buttons in a web browser, and demonstrated that simple mouse gestures could be used to decrease the time taken to execute these actions. 1.1. Gestures and Auditory Feedback The benefits of gestural interaction can be greatly increased when combined with auditory feedback: users utilise both auditory and visual senses when trying to classify and thus understand a ges- ture [5]. Pirhonen et al [6] designed the TOUCHPLAYER music player for a PDA. This allowed users to operate the player with gestures. Users were presented with auditory feedback in the form of earcons following a gesture being made, which confirmed the command had registered and what the command was. Testing demonstrated that gestures followed by clear auditory feedback were rapidly learnt. The particular use of auditory feedback in systems which utilise gestures is commonly seen as a method of enhancing user learnability [7]. Brewster et al. [3] developed a device which involved gestural interaction using head movements. Here, auditory enhancement was used as a precursor to command selection and not as feedback. A user would hear spatial earcons through headphones, and would make a nodding gesture towards the required command. Results from this study indicated that a limited menu of items showed in- creased usability when dynamic earcons were coupled with head gestures. 2. AUDIO-ENHANCED GESTURAL INTERACTION IN A WEB BROWSING INTERFACE There is limited literature dealing specifically with sonically en- hanced gestural interaction for web browsing. Moyle and Cock- burn [4] investigated forward and back gestures in a web browser (using a technique called marking menus - see [8] and [9]). The motivation was to reduce the time taken to carry out navigational operations in a web browser. This is similar to the motivation be- hind this project, except that we will be looking at user satisfaction rather than speed. Two important issues associated with designing gestural inter- action are [9, 10]: 1. The coordinates of a mouse gesture must change from the start to finish point by a minimum of 35 pixels, otherwise the gesture may be recognised as a simple mouse click. 2. A mouse gesture must be completed within 250 ms. in or- der to prevent it being recognised as a mouse click or a copy-and-paste-type standard manoeuvre. These issues manifest themselves in the browsing environment as unrecognised gestures, unwanted gestures, and a level of confusion over what is happening in the interface. It is in this area particu- larly that a level of auditory feedback might assist in reducing the number of gestural errors and confusion. In their study on imple- menting auditory enhancements for the MOSAIC browser, Albers & Bergman [11] undertook an informal evaluation into how this feedback affected users. The study was not related specifically to gestures but to mouse interacton in general, however it is still instructive. Albers & Bergman concluded that auditory feedback could be used to confirm user actions without interrupting the vi- sual field. If we relate these findings back to the issue of gesture recognition, we can use the auditory feedback to confirm to the user what they have just done without distracting them visually. This may also help to reduce confusion with gestures as the feed- back given can clarify the current situation In exploring the advantages of Auditory Display in a web en- vironment Maasø [12] suggests sound cues can provide feedback on actions about to be made, as well as already made. Maasø also states that sound is well suited to monitoring background activities while performing other actions. If these ideas are applied to the ICAD06-1 ICAD06 - 187