Human-Computer Synergies in Prosthetic Interactions John M. Carroll, Sooyeon Lee, Madison Reddie, Jordan Beck, Mary Beth Rosson Center for Human-Computer Interaction, Pennsylvania State University University Park, Pennsylvania 16802 USA jmcarroll@psu.edu, sul131@psu.edu, mbr5511@psu.edu, beckj@msoe.edu, mrosson@psu.edu Abstract. Remote sighted assistance provides prosthetic support to people with visual impairments (PVI) through internet-mediated conversational interactions. In these interactions, PVI broadcast live video to remotely-located, sighted people who engage in speech interactions with PVI to create prosthetic support. These interactions can be quite nuanced, creative, and effective. In this paper, we present a design investigation of remote sighted assistance (RSA) in which computer vision capabilities are integrated into the prosthetic interaction, supporting the human participants in various ways. Our study involved creating design scenarios to identify and concretize future possibilities in order to articulate and analyze design rationale for these scenarios, that is to say, strengths and challenges of RSA integrated with CV. We discuss implications for the design of the next generation of remote sighted assistance. Keywords: Computer Vision (CV); Design Envisionment Scenarios; Prosthetic Interactions; People with Visual Impairments (PVI); Remote Sighted Assistance (RSA) 1 Introduction Vision is an important tool for most people to understand the world around them, especially when significant objects are too far away to touch and either do not make sounds, or occur in noisy environments. For 50 years, computer vision technologies have been pursued to augment the experiences of people with visual impairments (PVI). Computer vision (CV) seeks to provide direct support to PVI by describing the spaces and things around them. In principle, it can leverage analysis of the PVI’s context and current goals. For example, prior work addressed grocery shopping [49], [88], [89], an everyday activity that is constrained and simplified by being carried out in a context of aisles and shelves, and an object taxonomy that is restricted to food and household items. If a computer vision assistive system is primed with a model of shopping activities, it should be able to use the model to better predict and respond to PVI goals and needs. A complementary type of visual prosthetic is remote sighted assistance (RSA), which relies on conversational interactions with a remote person who provides assistance as needed or requested. In these interactions, PVI broadcast a personal Interaction Design and Architecture(s) Journal - IxD&A, N.44, 2020, pp. 29 - 52 29