Modeling of video projectors in OpenGL for implementing a spatial augmented reality teaching system for assembly operations Carlos M. Costa 1 , Germano Veiga 1 , Armando Sousa 1 , Luís Rocha 2 , A. Augusto Sousa 3 , Rui Rodrigues 3 and Ulrike Thomas 4 Abstract—Teaching complex assembly and maintenance skills to human operators usually requires extensive reading and the help of tutors. In order to reduce the training period and avoid the need for human supervision, an immersive teaching system using spatial augmented reality was developed for guiding inexperienced operators. The system provides textual and video instructions for each task while also allowing the operator to navigate between the teaching steps and control the video playback using a bare hands natural interaction interface that is projected into the workspace. Moreover, for helping the operator during the final validation and inspection phase, the system projects the expected 3D outline of the final product. The proposed teaching system was tested with the assembly of a starter motor and proved to be more intuitive than reading the traditional user manuals. This proof of concept use case served to validate the fundamental technologies and approaches that were proposed to achieve an intuitive and accurate augmented reality teaching application. Among the main challenges were the proper modeling and calibration of the sensing and projection hardware along with the 6 DoF pose estimation of objects for achieving precise overlap between the 3D rendered content and the physical world. On the other hand, the conceptualization of the information flow and how it can be conveyed on-demand to the operator was also of critical importance for ensuring a smooth and intuitive experience for the operator. I. I NTRODUCTION Teaching humans new manufacturing skills or advanced assembly / maintenance operations can be a long and error prone process that requires reading extensive manuals and a period of tutoring in which they are trained by field experts. This teaching period can be shortened and done without the need of other workers by relying on immersive Human Machine Interface (HMI) teaching systems that are able to transmit the knowledge more effectively using step by step instructions containing text and video along with visual cues showing the work areas and pick / place locations tagged with contextual help. Moreover, when coupled with active perception systems that can detect the assembly objects and what the operator is doing, the teaching systems can also act as a supervisor, alerting the operator when a mistake is made or when a damaged component needs to be replaced. This 1 Authors with the Centre for Robotics in Industry and Intelligent Systems (CRIIS) of INESC TEC and with the Faculty of Engineering of the University of Porto, Portugal (emails: {carlos.m.costa, germano.veiga}@inesctec.pt and asousa@fe.up.pt) 2 Author with the Centre for Robotics in Industry and Intelligent Systems (CRIIS) of INESC TEC, Portugal (email: luis.f.rocha@inesctec.pt) 3 Authors with the Centre for Information Systems and Computer Graphics (CSIG) of INESC TEC and with the Faculty of Engineering of the University of Porto, Portugal (emails: {augusto.sousa, rui.rodrigues}@fe.up.pt) 4 Author with the Robotics and Human Machine Interaction Laboratory at the Technical University of Chemnitz, Germany (email: ulrike.thomas@etit.tu-chemnitz.de) approach to skill transfer using immersive HMIs along with dynamic feedback can speedup and improve the effectiveness of the training sessions while also giving continuous quality control, allowing to reduce the cost and time of product assembly, repair and maintenance. With these goals in mind, a spatial augmented reality teaching system was developed for projecting into the oper- ator workspace contextual assembly instructions that provide detailed information about the operations and tools that are required to assemble a given product. The main advantage of this approach is its ability to provide accurate 3D information directly into the environment and only when it is needed. For production lines that may receive custom products or have their employees rotating between workstations, a spatial augmented reality system can quickly and intuitively guide them throughout the assembly process. Moreover, the pro- posed system can also be used for coordinating tasks between human operators and robotic systems, because it is able to highlight 3D work areas or objects and provide visual cues for informing the operator what the robot will be doing next and where it will be working. In the following section, a brief overview of the augmented reality systems that were developed over the years will be given. Then in Section III, the mathematical modeling and calibration of video projectors will be discussed. Later on, Sec- tion IV will describe the immersive HMI that was developed. Given the lack of Computer Aided Design (CAD) models of the starter motor used for testing our system, Section V will describe how the 3D model was retrieved using a structured light 3D scanner. Then, Section VI will present the 6 Degrees of Freedom (DoF) object pose estimation system. Finally, Section VII will discuss the results of a training session while Section VIII will summarize the conclusions and present possible future work. II. RELATED WORK Augmented Reality (AR) interfaces [1] have a wide range of applications within the manufacturing industry, from the design [2], simulation and planning phase for fast prototyp- ing, to the training and guidance [3] of the operators that will be manufacturing, assembling and providing maintenance for the final product [4]. They offer an immersive way of exchanging information between a human operator and a robot / machine, allowing the development of cooperative assembly lines [5]. This immersive environment can be created with a wide range of devices, such as projectors, smart glasses / watches / phones, tablets, Virtual Reality (VR) headsets, 978-1-7281-3558-8/19/$31.00 c 2019 IEEE