Pose Estimation and Video Annotation Approaches for Understanding Individual and Team Interaction During Augmented Reality-Enabled Mission Planning Sue Kase (B ) , Vincent Perry, Heather Roy, Katherine Cox, and Simon Su DEVCOM Army Research Laboratory, Aberdeen Proving Ground, Aberdeen, MD 21005, USA {sue.e.kase.civ,vincent.p.perry7.civ,heather.e.roy2.civ, katherine.r.cox11.civ,simon.m.su.civ}@mail.mil Abstract. Two video analysis approaches (pose estimation and manual annota- tion) were applied to video recordings of two-person teams performing a mis- sion planning task in a shared augmented reality (AR) environment. The analysis approaches calculated the distance relations between team members and annotated observed behaviors during the collaborative task. The 2D pose estimation algo- rithm lacked scene depth processing; therefore, we found some inconsistencies with the manual annotation. Although integration of the two analysis approaches was not possible, each approach by itself produced several insights on team behav- ior. The manual annotation analysis found four common team behaviors as well as behavior variations unique to particular teams and temporal situations. Compar- ing a behavior-based time on task percentage indicated behavior-type connections and some possible exclusions. The pose estimation analysis found the majority of the teams moved around the 3D scene at a similar distance apart on average with similar variation in fluctuation around a common distance range between team members. Outlying team behavior was detected by both analysis approaches and included: periods of very low distance relations, infrequent but very high dis- tance relation spikes, significant task time spent adjusting the HoloLens device during wearing, and exceptionally long task time with gaps in pose estimation data processing. Keywords: Augmented reality · Mission planning · Pose estimation 1 Introduction The proliferation of immersive technologies (augmented reality (AR) and virtual reality (VR) systems) has accentuated their potential utilization across a wide range of oper- ational situations from strategic planning to the tactical edge. However, little is known about individual performance and especially team collaboration associated with using immersive technologies. Researchers at DEVCOM Army Research Laboratory (ARL) have been using Microsoft HoloLens devices to investigate AR technology for support- ing collaborative mission planning. Shared AR spaces, such as simulated operational © Springer Nature Switzerland AG 2021 J. Y. C. Chen and G. Fragomeni (Eds.): HCII 2021, LNCS 12770, pp. 32–46, 2021. https://doi.org/10.1007/978-3-030-77599-5_3