Interservice/Industry Training, Simulation, and Education Conference (I/ITSEC) 2015 2015 Paper No. 15348, Page 1 of 12 Safe Testing of Autonomous Systems Performance David Scheidt, Robert Lutz, William D’Amico, Dean Kleissas, Robert Chalmers, Robert Bamberger, Johns Hopkins University Applied Physics Laboratory 11100 Johns Hopkins Road, Laurel, MD david.scheidt@jhuapl.edu, robert.lutz@jhuapl.edu, william.d’amico@jhuapl.edu, dean.kleissas@jhuapl.edu, robert.chalmers@jhuapl.edu, robert.bamberger@jhuapl.edu ABSTRACT The role of unmanned platforms is rapidly expanding across a wide range of defense and homeland security missions. Currently operational unmanned vehicles are “tele-operated”, using a command and control link to a remotely located pilot. However, operational complexity, operational pace, and a need to function in communication denied environments necessitate a trend toward autonomous unmanned vehicles. Autonomous systems that make independent decisions in complex engagements, such as the Navy’s Autonomous Aerial Cargo Unmanned System, are currently under development and will require development and operational testing within the next 3-5 years. Testing of autonomous systems presents some unique and vexing challenges. For instance, the infinite number of variations of test conditions that can exist to stimulate autonomous behaviors and the complexity of the interactions that can occur among multiple autonomous systems combine to make comparative measurement of autonomous system performance extremely difficult. Also, the inherent unpredictability of decision making by autonomous systems may result in decisions that are considered unsafe by managers of live test ranges. Advanced test and evaluation techniques that focus on the unique challenges of autonomy represent a clear and increasing need within the DoD. The Safe Testing of Autonomy in Complex, Interactive Environments (TACE) Program is a research initiative to develop an advanced test infrastructure that can measure the performance of autonomous systems operating in complex Live-Virtual-Constructive (LVC) environments while ensuring that the autonomous system does not violate range safety policy. This paper will provide an overview of the TACE hardware and software architecture and will highlight the LVC testing that has been performed at the Aberdeen Test Center to validate TACE capabilities. A discussion of anticipated transition activities with DoD partner programs will also be provided. ABOUT THE AUTHORS DAVID SCHEIDT is a member of the Principal Professional Staff at Johns Hopkins University’s Applied Physics Laboratory where he conducts research on distributed intelligent control systems. Mr. Scheidt has 30 years of experience in the research and development of distributed information management systems, robotics, artificial intelligence and process control systems. Throughout his career Mr. Scheidt has conducted research in concert with his development efforts, publishing over 50 peer reviewed publications for research funded by the National Computer Security Center (NCSC), ONR, NASA, DISA, OSD NII and the US Army. Mr. Scheidt currently conducts autonomous systems and intelligent controls research and is the principal investigator on research initiatives that focus on the intelligent diagnosis, reconfiguration and planning of ship auxiliary systems, spacecraft and unmanned vehicles. ROBERT LUTZ is a principal staff scientist at The Johns Hopkins University Applied Physics Laboratory in Laurel, MD. His background includes 34 years of practical experience in the development, use, and management of models and simulations across all phases of the DoD systems acquisition process. He currently serves as the LVC and testing lead for the TACE project and as the Airspace Integration Modeling and Simulation (M&S) lead for the Navy’s Triton Program. Mr. Lutz also serves as the Chair of the Simulation Interoperability Standards Organization (SISO) Board of Directors, serves on the Tutorial Board and Fellows Committee at the Interservice/Industry Training, Simulation and Education Conference (I/ITSEC), and is a guest lecturer on M&S-related topics in The Johns Hopkins University Whiting School of Engineering. WILLIAM D’AMICO, Ph.D., is a principal staff scientist at The Johns Hopkins University Applied Physics Laboratory in Laurel, MD. His background includes over 40 years of research, development, and program management in gun-launched projectile systems, micro-electromechanical systems, and unmanned aerial vehicles.