West Point Autonomous Vehicle Research and Design: Convolutional Neural Networks and Image Classification Edward Kang 1 , Sean Min 1 , Will Anderson 2 , Reed Burton 2 , Jarred Fassett 2 , Samuel Pool 2 , Mitchell Stiffler 2 , Preston Draelos 3 , Chris Little 3 , Austin Morock 3 , Courtney Razon 1 Daniel Gonzalez 2 , Kathryn Pegues 1 , and Joseph Cymerman 3 1 Department of Systems Engineering 2 Department of Electrical Engineering and Computer Science 3 Department of Civil and Mechanical Engineering United States Military Academy, West Point, NY Corresponding Author: Edward.k.kang.mil@mail.mil Author Note: The authors of this article participated in a year-long senior research project in the Departments of Systems Engineering (SE), Civil and Mechanical Engineering (CME), and Electrical Engineering and Computer Sciences (EECS) at the United States Military Academy (USMA). The research underpinning this paper was conducted under the supervision of Dr. Daniel Gonzalez, Major Courtney Razon, Lieutenant Colonel Kathryn Pegues, Major Joseph Cymerman, Major Mark Lesak, Dr. Peter Hanlon, Mr. Dominic Larkin, and Mr. Nicholas Livingston. Upon graduating in May 2020, the authors commissioned into the United States Army as Second Lieutenants, serving in the Aviation, Cyber, Engineer, Field Artillery, Infantry, and Military Intelligence branches. Abstract: Technological development within the domain of autonomous driving systems remains an important area of focus for government, industry, and academia. Large corporations are continuing research and development in unmanned delivery platforms; this technology holds the potential to substantially decrease costs and increase efficiency in supply chain management. For the military, autonomous systems show the potential to increase capability and improve Soldier safety. The West Point Autonomous Vehicle Research and Design (AVRAD) team developed an autonomous system, capable of competing in the 2020 Intelligent Ground Vehicle Competition (IGVC)--Self Drive Competition. IGVC leadership decided to cancel the 202 IGVC due to the COVID-19 pandemic. However, for the past 27 years, this international competition challenged teams to think creatively about evolving technologies of vehicle sensors, robotics, and system integration. During the event, teams race their autonomous systems through a course which tests the entry’s lane following ability and ability to avoid obstacles. While driving, the system builds a map of the immediate area by using data gathered from three front facing and two rear facing Mako G-319C cameras. Additionally, a Velodyne HDL-64E LiDAR senses and identifies each type of obstacle or critical image such as a stop sign. The images gathered are relayed to a Convolutional Neural Networks (CNN). The CNN algorithm classifies each image as a specific item, whether it is a traffic sign, person, or an entirely different obstacle. Finally, the vehicle management system uses the CNN’s classification output to map the immediate surroundings. With an established map, the system then determines the most appropriate action or pathway. This paper will explain, in detail, how the AVRAD team selected, tested, and improved a CNN capable of quickly and accurately detecting and recognizing common street side objects along its path. 1. Background The Department of Defense (DoD) remains interested in evolving robotic technology as it has the potential to increase soldier safety in volatile, uncertain, complex, and ambiguous environments. The U.S Army’s Ground Vehicle Systems Center (GVSC), the leading development agency in robotic development in the Army, develops, integrates, and sustains technological solutions for manned and unmanned ground vehicle systems. To incentivize research in this domain, the GVSC sponsors the Intelligent Ground Vehicle Competition (IGVC—Self Drive) annually in Rochester, Michigan. The focus of IGVC—Self Drive is to encourage teams from various undergraduate and graduate programs to gain experience in developing autonomous systems capable of navigating urban environments. The West Point Autonomous Vehicle Research and Design (AVRAD) team’s physical system is a Polaris GEM E2 vehicle outfitted with off-the-shelf components capable of achieving autonomous driving. With a functioning vehicle, the team focused achieving a critical component of autonomous driving- the ability to detect, classify, and react to obstacles. Proceedings of the 2020 Annual General Donald R. Keith Memorial Capstone Conference West Point, New York, USA April 30, 2020 A Regional Conference of the Society for Industrial and Systems Engineering ISBN: 97819384961-8-9 200