On the Role of Latent Design Conditions in Cyber-Physical Systems Security Sylvain Frey, Awais Rashid, Alberto Zanutto, Jerry Busby, Karolina Follis Security Lancaster Research Centre, Lancaster University, United Kingdom {s.frey, a.rashid, a.zanutto, j.busby, k.follis}@lancaster.ac.uk ABSTRACT As cyber-physical systems (CPS) become prevalent in ev- eryday life, it is critical to understand the factors that may impact the security of such systems. In this paper, we present insights from an initial study of historical security incidents to analyse such factors for a particular class of CPS: indus- trial control systems (ICS). Our study challenges the usual tendency to blame human fallibility or resort to simple ex- planations for what are often complex issues that lead to a security incident. We highlight that (i) perception errors are key in such incidents (ii) latent design conditions – e.g., improper specifications of a system’s borders and capabilities – play a fundamental role in shaping perceptions, leading to security issues. Such design-time considerations are par- ticularly critical for ICS, the life-cycle of which is usually measured in decades. Based on this analysis, we discuss how key characteristics of future smart CPS in such industrial settings can pose further challenges with regards to tackling latent design flaws. Categories and Subject Descriptors Software Engineering [D.2.10]: Design Keywords cyber-physical systems, industrial control systems, risk, per- ception, design 1. INTRODUCTION The role and impact of users on the security of regular IT systems is a common matter of study in literature. For in- stance [1, 2] investigate how users’ psychological and cognitive biases affect security features and discuss how better sys- tem designs should account for these human characteristics. These works identify users’ perception as a critical aspect, including non-malicious behaviours becoming a threat [2]. Industrial control systems (ICS) differ from pure IT sys- tems that are the focus of such works. Firstly, ICS are Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from permissions@acm.org. SEsCPS’16, May 16 2016, Austin, TX, USA c  2016 ACM. ISBN 978-1-4503-4171-4/16/05. . . $15.00 DOI: http://dx.doi.org/10.1145/2897035.2897036 cyber-physical systems (CPS) that combine regular software systems with physical ones that control and operate various sensors and actuators impacting their environment. Secondly, ICS have a greater diversity of human roles around them, e.g., operator, technician, maintainer, engineer, manager, etc. instead of mere“end-user”. Finally, and most importantly, ICS’ typical life cycle is counted in decades: the interval be- tween the design and the end of life of a system can span over 50 years. Such long periods leave little room for evolution, due to stringent availability and safety requirements in a number of infrastructures, some of them critical – power grid, water supply, rail network, etc. The design of such systems is therefore a fundamental step with long-term consequences. Of course, risk perception has been studied in industrial contexts [4, 14] and safety has been a central concern in such settings [6]. However, the increased connectivity of ICS and the emergent smart CPS settings pose challenges not just for safety but for security – with security lapses inadvertently impacting on safety (as our case studies show). To our knowledge, ours is the first work to undertake a socio- technical analysis of perception errors underpinning security issues in CPS. In this paper, we start by considering the role of ICS oper- ators’ perception during security incidents. We investigate a corpus of 6 case studies to analyse how operators perceive the system and its various parts before and during an inci- dent. Our investigation reveals that perception errors are central to all the case studies we investigated. We identify and classify perception errors and analyse the possible causes and conditions behind them. We show that, beyond individ- ual operator mistakes, latent design conditions [13] play a fundamental role in shaping perceptions, leading to security issues. Our study offers two key insights: 1. We challenge the idea that humans are necessarily the weak link via which most incidents occur. We show that latent design condition are a key factor that shapes operators’ perception, leading to operational mistakes and incidents. Our classification of perception errors provides additional insights regarding different types of latent design flaws, in terms of system borders, capabilities, observability and controllability. 2. We discuss how fundamental characteristics of future smart CPS deployed in such settings can further compli- cate the early identification and management of latent design flaws.