Egocentric Place Recognition for AR Wayfinding Assistive Technology Charbel Azzi, John S. Zelek University of Waterloo Waterloo, ON, Canada cazzi/jzelek@uwaterloo.ca Abstract Wayfinding is something we take for granted whether it is from within our vehicles or by foot. We usually know where we are by our position to some landmarks in the context of some destination. GPS technology has helped augment our wayfinding in unfamiliar surroundings. If someone is perceptually (i.e., blind) or cognitively (i.e., dementia) chal- lenged, then precise localization - more precise than a GPS - is essential for simple tasks like knowing where the curbs are or when crossing a road. The advent of geocoded im- age streetview data that provides a database for us to geo- reference to from an ego-centric worn camera when GPS is not accurate enough or non existent such as indoors. The method of augmentation of the experience does not neces- sarily lend itself to being visual with the lack of visual or cognitive senses. We have explored haptic augmentation due to its primitive reactive responses. Most of our cur- rent focus now are on the challenges associated with place recognition including the building of a map from streetview imagery and the indexing and searching with this geocoded database of images. The tasks include navigation, obstacle avoidance and accurate geo-referencing. 1. Introduction The number of elderly people in the world is increas- ing in proportion to the total number of people. As people get older, their cognitive and perceptual faculties decline in functionality and sometimes fail entirely; for example, man- ifesting themselves in disease or illness such as blindness or Alzheimers disease. Approximately 14 million people in North America are affected by blindness and 7 million peo- ple suffer from Alzheimers. Technology can help alleviate issues of confinement, security and safety as well as em- powering people who feel constrained by their condition. One such way is to ensure that they are still able to conduct their daily activities by providing them with technological navigational tools. The ability to perceive the local and immediate environ- ment for daily living activities is needed by all people. Peo- ple with visual impairments as well as people with cogni- tive impairments require technology that identifies obstacles in their immediate path both inside and outside of build- ings and assists in navigational tasks. Unanticipated factors such as construction work, streets and walkways in need of repair, garbage placement, poor weather, irregular traffic, other pedestrians and bicycles can make the basic challenge of navigating streets and sidewalks difficult. Improved lo- calization as well as obstacle detection and avoidance capa- bilities (that augment existing aids such as the long cane or guide dog) will support safe independent travel, increasing access to school, work and the community overall as well as increasing self-esteem and independence. We have designed, built and tested a tactile navigational belt that translates navigational orientational information (from a GPS positional fix differentiated towards a desired goal location) into tactile nudges along the waist. A GPS can tell people where they are and how to get somewhere but the interface can be complicated for the elderly. Field trials with persons with dementia demonstrated the success of the tactile interface [?], [?] (See Figure ??. It appeared that the nudging of the waist was almost like a sixth sense in that no or very little cognitive processing was needed, almost a reactive type of response. We also reported upon a similar device used with people who were blind where a hand tactile device was used to convey orientation to avoid obstacles as sensed by a worn stereo vision camera [?] These two tactile devices show that tactile feedback as a local or global orientation guide is an effective feedback mechanism. GPS as a localization device is only accurate to a few meters (i.e., 3 m) and is not available indoors. Also the environment may contain obstacles such as bumps in the sidewalks. An egocentric camera can provide a finer lo- calization fix when fused with GPS if available. Also, the sensed images can indicate obstacles and hazards that are to be avoided. Route navigation in spaces that are too large to view from a single perspective is achieved through comparison of perceptual information about our orientation and motion 1