Using affective human–machine interface to increase the operation performance in virtual construction crane training system: A novel approach Iman Mohammad Rezazadeh a, ⁎, Xiangyu Wang b , Mohammad Firoozabadi a,c , Mohammad Reza Hashemi Golpayegani a,d a Department of Biomedical Eng., Science and Research Branch, Islamic Azad University, Tehran, Iran b Senior Lecturer, Faculty of Built Environment, The University of New South Wales, Australia, and International Scholar at Department of Housing and Interior Design, Kyung Hee University, Seoul, Korea c School of Medical Sciences, Tarbiat Modares University, Tehran, Iran d Department of Biomedical Eng., Amir Kabir University of Technology, Tehran, Iran abstract article info Article history: Accepted 24 August 2010 Available online 26 November 2010 Keywords: Virtual Reality Facial bioelectric-signals Affective computing Affective measures Construction training In the construction industry, some progress have been achieved by researchers to design and implement environments for task training using VR technology and its derivatives such as Augmented and Mixed Reality. Although, these developments have been well recognized at the application level, however crucial to the virtual training system is the effective and reliable measurement of training performance of the particular skill and handling the experiment for long-run. It is known that motor skills cannot be measured directly, but only inferred by observing behaviour or performance measures. The typical way of measuring performance is through measuring task completion time and accuracy, but can be supported by indirect measurement of some other factors. In this paper, a virtual crane training system has been developed which can be controlled using control commands extracted from facial gestures and is capable to lift up loads/materials in the virtual construction sites. Then, we integrate affective computing concept into the conventional VR training platform for measuring the cognitive load and level of satisfaction during performance using human's forehead bioelectric-signals. By employing the affective measures and our novel control scheme, the designed interface could be adapted to user's affective status during the performance in real-time. This adaptable user interface approach helps the trainee to cope with the training for long-run performance, leads to gaining more expertise and provides more effective transfer of learning to other operation environments. The detailed methodology of the affective control is presented in the paper. The results and future applications of the proposed method for disabled users, especially from neck down are discussed. © 2010 Elsevier B.V. All rights reserved. 1. Introduction Construction equipment operators operate one or several various types of powered construction equipment. Virtual technologies afford new opportunities for effectively training novices with lower cost and fewer hazards. Seidel and Chatelier [1] have suggested, for example, that the use of virtual environments (VEs) may be “training's future”. Virtual environments can be especially valuable where training in real-world situations would be impractical because a real field scenario may be unduly expensive, logistically difficult, dangerous, or too difficult to control. Compelling virtual environments could lead human participants to feel somehow present, for purposes of training. Virtual environments involve structure of a human participant existing within some kind of an artificial interaction environment. Part of the components involved in the interactive virtual training environment are simulated, the operator nevertheless can experience a similar sense of being present and interacting with real/virtual objects via visual, auditory or force displays. Virtual environments can also assist with the delivery of equipment operation training during inclement weather conditions and novices have much more time to practice their skills without the pressure of costs. Research has shown that the key to acquiring the necessary motor skills to control complex systems, such as a backhoe excavator, is hands-on and coached training [2]. This approach is envisaged to facilitate progress along what is a steep learning curve and enable effective rehearsal of future operations in actual construction sites. The promise of effectiveness is supported by evidence from mental health research revealing that a virtual experience can evoke the same reactions and emotions as a real experience [3]. These technologies should become the bridge connecting the ideal training objective to the current reality of training programs. Research efforts in VR training for construction have been predominantly focused on a proof-of-concept level of implementing Automation in Construction 20 (2011) 289–298 ⁎ Corresponding author. E-mail addresses: I_rezazadeh@yahoo.com, i.rezazadeh@srbiau.ac.ir (I.M. Rezazadeh). 0926-5805/$ – see front matter © 2010 Elsevier B.V. All rights reserved. doi:10.1016/j.autcon.2010.10.005 Contents lists available at ScienceDirect Automation in Construction journal homepage: www.elsevier.com/locate/autcon