Journal on Multimodal User Interfaces https://doi.org/10.1007/s12193-018-0261-7 ORIGINAL PAPER Real-time eye blink and wink detection for object selection in HCI systems Hari Singh 1,3 · Jaswinder Singh 2 Received: 17 May 2017 / Accepted: 21 January 2018 © Springer International Publishing AG, part of Springer Nature 2018 Abstract This paper presents an approach for real-time detection of three types of eye blinks: eye blink (blinking both eyes simulta- neously), left and right winks. The process of blink detection has been divided into four parts viz. face localization in facial images acquired through a video camera, eye pair localization, pixels’ motion analysis using optical flow technique, and classification of eye blinks. Blink detection has been performed using a video camera and MATLAB software with image processing and computer vision toolbox. The algorithm takes about 60 ms time for processing a frame and 250 ms time for confirmation and classification of the detected blink. An experiment was conducted to evaluate the performance of the proposed approach in which 10 users voluntarily participated. The performance of the proposed method has been tested under two lighting conditions: natural lighting conditions and controlled lighting conditions. Also, the performance has been tested by varying the distance of the user from the camera. Here, it is observed that the system gives best performance when used under controlled lighting conditions and the user sitting at a distance of about 0.5 m. Accuracy of the proposed approach has been found to be 96, 92 and 88% for detection of eye blink, left wink and right wink, respectively. The proposed method has also been tested on ZJU dataset where it has given precision, detection accuracy and false alarm rate of values 94.11, 91.2 and 1.54%, respectively. The proposed system has been used and evaluated for performing various mouse analogous functions using eye blinks and winks. It has given an accuracy of 90, 80 and 90% in performing left click, double click, and right click operations, respectively. Keywords Real-time eye blink detection · Target selection · Analogous mouse operations · HCI systems 1 Introduction Eye blink detection is used in different applications e.g. object selection in HCI systems [1], driver fatigue detection [2,3], or liveness detection [4]. Object selection is a very sig- nificant part of a human–computer interaction system. Two commonly used selection triggers are the key trigger, where B Hari Singh harisdhillon@gmail.com Jaswinder Singh j_singh73@rediffmail.com 1 I.K. Gujral Punjab Technical University, Jalandhar, India 2 Department of Electronics and Communication Engineering, Beant College of Engineering and Technology, Gurdaspur, India 3 Department of Electronics and Communication Engineering, DAV Instituteof Engineering and Technology, Kabir Nagar, Jalandhar 144008, India user presses a key on the keyboard [5], and a dwell time trig- ger whereby users fixate on an object for a period of time (dwell time) exceeding a predetermined threshold to trigger a selection [6–11]. Other types of selection triggers such as eye blinking [7,12–18], on–off screen buttons, gaze gestures, pEYEs, Dashers [19], EMG signals [20,21], mouth open- ing click [22], brows up clicking [16,22], tooth clicker [23], clicking with smiling [24], and antisaccade clicking [8], etc have also been cited in the literature. These object selection techniques provide a practical method of interacting with computers, but they fail to serve the most severely motor impaired [13] where blinking may be the only choice for object selection. So, object selection by eye blinks becomes the obvious choice for persons suffering from Amyotrophic Lateral Sclerosis (ALS) and other locked-in syndromes. Two important methods of eye blink detection cited in the literature are electrooculography (EOG) [25–27] and videooculography (VOG) [1,12–16,18,28–30]. In case of EOG, Ag/AgCl electrodes are placed around the eyes 123