J Multimodal User Interfaces DOI 10.1007/s12193-017-0252-0 ORIGINAL PAPER Studying gesture-based interaction on a mobile augmented reality application for co-design activity Leman Figen Gül 1 Received: 30 November 2016 / Accepted: 6 November 2017 © Springer International Publishing AG, part of Springer Nature 2017 Abstract Multi-touch user interfaces (MTUIs) of mobile devices can represent a valuable tool for enhancing co-design practice. In particular, mobile augmented reality (AR) tech- nology that enhanced with MTUIs can offer new possibilities to designers to work and collaborate. This paper describes a series of user studies that took place in the Department of Architecture, at the Istanbul Technical University in January 2016. The users tested the developed AR application. The task of the participants was to design mass volumes of build- ings in the given contexts. In the first study, the designer worked solo using analogue tools and two different MTUIs of the AR applications. In the second study, the designers collaborated using analogue tools and an enhanced mobile augmented reality environment. The goal of the study is to understand the affordances of the interfaces and the changes onto designers’ behaviour when they are using the mobile AR applications. The particular focus of the paper is on the characterization of the co-design cognition and inter- action behaviour of the designers. The collected data from the empirical studies is analysed with the protocol analysis method using a coding scheme. The results show that the studied AR interfaces afford different modelling actions and design behaviour. The design context was broad conceptual and high-level; the intensity of the actions was also different in the AR sessions. We consider that this knowledge would be informative for the development of the innovative augmented reality applications for supporting the design activity and be guidance for the further developments. B Leman Figen Gül fgul@itu.edu.tr 1 Faculty of Architecture, Istanbul Technical University (ITU), Istanbul, Turkey Keywords Mobile augmented reality · Affordances of touchscreen · Collaborative design · Protocol analysis 1 Introduction Over the last decade, with the proliferation of the mobile communication technologies, there has an increase in the development of the applications and devices of Augmented Reality (AR). In general, AR technology includes a combi- nation of the real and the virtual by providing 3D real time interactivity [1]. The early version of the AR technology is mainly seen as a representation tool for the evaluation of the design proposal [2, 3]. The recent mobile AR technol- ogy has the potential to offer new opportunities to designers as the new design platform that the visual and the physical models can be superimposed for the real time design and development and visual analysis of the design idea. Thus, the AR technology can provide the shared virtual model for co-designers to discuss and interact with it. Gestures have great importance in the design context (see [4] for a review). Multi-touch user interfaces (MTUIs) offer users the interaction with the computer applications through a set of gestures [5]. With the affordability of the mobile devices that have the MTUIs, such as phones and tablets, more intuitive and direct interaction with the digital world becomes possible. Although there are some applications of the gesture interaction with the visual objects in the 2D space, the gesture based interaction with the 3D space in a co-design context is a new area. There are many classifications applied to gestures such as semiotic, ergotic and epistemic (see [6] for more details on the topic). In this paper, we consider that the gestural equivalent of the interfaces of direct manipulation are those which interaction occur with tapping, pointing and pinching 123