An Egocentric Computer Vision based Co-Robot Wheelchair* Haoxiang Li, Mohammed Kutbi, Xin Li, Changjiang Cai, Philippos Mordohai, Gang Hua Abstract— Motivated by the emerging needs to improve the quality of life for the elderly and disabled individuals who rely on wheelchairs for mobility, and who might have limited or no hand functionality at all, we propose an egocentric computer vision based co-robot wheelchair to enhance their mobility without hand usage. The co-robot wheelchair is built upon a typical commercial power wheelchair. The user can access 360 degrees of motion direction as well as a continuous range of speed without the use of hands via the egocentric computer vision based control we developed. The user wears an egocentric camera and collaborates with the robotic wheelchair by convey- ing the motion commands with head motions. Compared with previous sip-n-puff, chin-control and tongue-operated solutions to hands-free mobility, this egocentric computer vision based control system provides a more natural human robot interface. Our experiments show that this design is of higher usability and users can quickly learn to control and operate the wheelchair. Besides its convenience in manual navigation, the egocentric camera also supports novel user-robot interaction modes by enabling autonomous navigation towards a detected person or object of interest. User studies demonstrate the usability and efficiency of the proposed egocentric computer vision co-robot wheelchair. I. I NTRODUCTION According to the National Institute of Child Health and Human Development (NICHD), 2.2 million people in the United States depend on a wheelchair for day-to-day tasks and mobility [1]. Most of them are elderly or disabled individuals, to whom independent mobility is very impor- tant. However, operating the existing manual or powered wheelchairs could be difficult or impossible for many in- dividuals [30]. Even with powered wheelchairs, people with severe upper body motor impairment may not have enough hand functionality to use the joystick. To accommodate these severely disabled individuals and support their independent mobility, researchers developed a number of alternative wheelchair controls [16], [6], [5], [22], [29], [31], [8], [19], [3], [25]. These hands-free control could improve the life quality of those individuals as well as people with good hand functionality but want to drive powered wheelchairs while keeping their hands free from holding the joystick. In early years, to support hands-free wheelchair driv- ing, researchers proposed special equipment or specifically *This work is supported in part by the National Institute Of Nurs- ing Research of the National Institutes of Health under Award Number R01NR015371. Haoxiang Li is with Adobe Research, San Jose, CA 95110. He contributed to this paper while he was with the Department of Computer Science, Stevens Institute of Technology, NJ 07030, USA. Mohammed Kutbi, Xin Li, Changjiang Cai and Philippos Mordohai are with the Department of Computer Science, Stevens Institute of Technology, Hoboken, NJ 07030, USA. Gang Hua is with the Department of Computer Science, Stevens Institute of Technology, Hoboken, NJ 07030, USA and Microsoft Research Asia, Beijing, China. designed wheelchairs such as the sip-n-puff control, head control and chin control [16]. In the sip-n-puff system, the user gives commands by applying different pressure on a pneumatic tube by “sipping” and “puffing”. It is a solution for severely disabled users as it requires minimal efforts in moving the upper body. However, it requires the user to switch between deep and shallow inhales and exhales, which affects the user’s natural breathing rhythms. Therefore it may not be comfortable for a long time. Besides, the user cannot communicate with others while operating this control. Chin-control or head-control can be a feasible solution when the user has good head movement ability. In a head- control system, switches are usually mounted in the headrest, and they are operated by head movement. In a chin-control system, the design allows the chin to sit in a cup shaped joystick handle and to control the handle by neck flexion, extension, and rotation. These two control schemes require the users to frequently move their neck and head and apply force on tactile sensors. More recent works include the tongue-based human-machine interface [18], [19], brain-controlled wheelchairs [17], [4] and voice-controlled wheelchairs [23], [7]. The tongue-based control is motivated by the observation that the tongue has rich sensory and motor cortex representation. Some work along these lines uses inductive devices installed in the user’s mouth and on the user’s tongue to provide multi-directional control of the wheelchair [18]. Non-invasive tongue-based control has also been developed. For example, Mace et al. [19] present a system to capture the tongue-movement induced ear pressure for wheelchair control. The tongue-based solution also has the drawback that it interrupts the user’s communication with other people. The brain-controlled wheelchairs have attracted a lot of attention due to their wide applicability [17], [4]. The brain- computer interface (BCI) is based upon the fact that the electrical activity of the brain can be monitored using an array of electrodes placed on the scalp. The user can drive the wheelchair without physically operating any mechanical device. For example, Carlson et al. [4] present a BCI system in which the user controls the wheelchair by performing a motor imagery task. The user is required to imagine the kinaesthetic movement of the left hand, the right hand or both feet. The signals can then be captured and classified into three classes and generate different commands to drive the wheelchair. The brain-controlled wheelchair has great potential in helping severely disabled individuals to obtain independent mobility but it demands the user’s full attention in generating the motion commands, which may not be a