LiLiPUT: Lightweight Lab Equipment for User Testing in Telecommunications Peter Fröhlich; Peter Reichl; Raimund Schatz; Lynne Baillie; Wolfgang Weinberger; Florian Hammer Telecommunications Research Center Vienna (ftw.), Donau-City-Straße 1, 1220 Wien, Austria {froehlich, reichl, schatz, baillie, weinberger, hammer}@ftw.at User testing in mobile HCI is still mostly conducted in stationary labs, even though the importance of field data is widely acknowledged. Starting from related work and a set of six key requirements for more successful and widespread field-based user testing, we present our Lightweight Lab Equipment for User Testing in Telecommunications environment (“LiLiPUT”). Keywords: LiLiPUT, mobile usability, mobile HCI, methodology, laboratory equipment 1. INTRODUCTION Recent research has clearly demonstrated that standard stationary lab testing is not sufficient for mobile usability evaluation and HCI research. First, users engaging in mobile situations tend to have focuses other than the device in front of them, a situation quite different compared to an indoor laboratory setting [1]. Thus, in order to obtain ecologically valid results, usability tests for mobile devices should be conducted in typical usage situations outside the lab. Second, whenever mobile context factors matter for design or research questions (e.g. amount of interruptions or the input modality depending on the situation), field studies need to be conducted [2]. Third, some mobile interaction types such as spatially aware services simply cannot be evaluated satisfactorily indoors. Different kinds of field research methods have been proposed in order to better understand and better match the nature of mobile contexts: ethnographic field studies, diaries and experience sampling [3]. However, such studies “in the wild” are still quite rare; the vast majority of mobile usability tests and also many HCI research experiments for mobile applications are still carried out in stationary labs. One important reason for this is the additional effort required for test execution in an outside environment and the complexity of (manually prepared) data analysis required for most field research methods. Furthermore, the amount and reliability of collected data (e.g. video, audio, and observer annotations) tends to be lower than in stationary user studies. 2. REQUIREMENTS FOR FIELD-BASED USER TESTING EQUIPMENT We have defined the following lab equipment requirements to enable more successful and widespread field-based user testing of telecommunications systems: 1. Portability (adequate size, power consumption for mobile equipment, etc.) 2. Data richness and accuracy: The collected data should be of a comparable level (e.g. comprehensiveness and accuracy) to that of a stationary lab (recorded handheld screen capture, video and audio data to document the test situation) 3. Workflow efficiency: The required workload for the research or consultancy team should not be higher than that of a traditional lab study. The data organisation and analysis should not consume more labour resources than when using a traditional lab. 4. Mobile contextual data capture (weather conditions, surrounding physical environment, gestures and emotional expression, etc.) 5. Natural and seamless interaction: Users should be able to move normally e.g. for example make telephone calls whilst engaging with the mobile device without any hindrances. To this end, they also should not feel uneasy or inhibited due to cables, obtrusive cameras or the surrounding team of observers 6. Coverage of all relevant mobile telecom application areas (not only mobile HCI, but also human-human communication and related QoS (Quality of Service) issues) 3. RELATED WORK Mobile HCI research has only recently tried to meet some of the requirements outlined above. The most advanced solution in this regard was proposed by Roto et al [4]. In essence, 1 microphone signal and 4 camera signals are mixed and recorded on a DV recorder located in the user’s backpack. The cameras capture the mobile device display, the user’s face, the user’s front (from the user’s shoulder), as well as the user from the perspective of the observer (from a wireless camera on the observers shoulder. The proposed solution fulfils requirements 1-3 to a high degree, enabling a reasonable amount of mobility, data richness and accuracy, as well as a very high workflow efficiency (there is only one observer and the data is ready to be coded after the test). Furthermore, the video of the user’s front partly helps to meet requirement 4, i.e. capturing mobile context. However, the obtrusive cables and cameras, mounted on the mobile device, may considerably hinder naturalness and seamlessness (req.