Probabilistic Model of Control-Flow Altering based Malicious Attacks (Poster submission) Sergey Frenkel Federal Reserach Center "Computer Science and Control " Russian Academy of Sc., Moscow, Russia, fsergei51@gmail.com Introduction. The system designers need in various design tools which could help them both for estimation of possible threats to the security and select one or another ways of their neutralization. There are many approaches to the evaluation (verification) of the degree of protection of programs against possible attacks. First of all, this is fault Injection (FI) simulation techniques [1]. Main drawback of the FI is necessity to have different expensive software that can be not used to solve other design problems, in particular for functional verification and testing. Also, due to the similarity between system failures because of intentional attacks and those due to accidental component failures, reliability/availability-like models to evaluate system survivability are used in the security design [2]. But they are based on Continuous Time Markov Chain (CTMC), identification of which deals with some technical difficulties. This paper considers a probabilistic approach to estimation of security risks of the programs due to malicious attacks which try to change the control flow of the program to corrupt the program behavior, the system calls sequence in particular. It is shown the possibility to use a Markov model with two absorbing states defined on direct product of the spaces of two finite state machines ( FSM), one of which is a program finite automaton model that is running under normal conditions, and second is the same FSM in which at some point in time (depending on the considered temporal discreteness) there was a failure due to external attacks (e.g., within the time of a single operation, or a program's block execution). Previously this model was suggested for hardware fault-tolerance analysis [6]. However, in contrast to the previously considered model, in which the effect of an erroneous state change was considered as a result of the damage of one or another bit of the status codeword, here we consider an altering of the program control flow (or system calls sequence ) as a cause of the attack malicious effect. Model of Program under Attacks. The application program model considered is the Finite State Machine (FSM) of Mealy type, corresponding to the algorithm implemented by this program. This FSM can be built either from a program source or from system calls sequences. We consider the attacks effect (that is the malicious codes action) as the control-data attack which alter the target program’s control data, say, as data that are loaded to processor program counter at some point in program execution. © Springer International Publishing AG 2017 O. Strichman and R. Tzoref-Brill (Eds.): HVC 2017, LNCS 10629, pp. 2 https://doi.org/10.1007/978-3-319-70389-3_22 49–252, 2017.