This article has been accepted for inclusion in a future issue of this journal. Content is final as presented, with the exception of pagination. IEEE SYSTEMS JOURNAL 1 Autonomic Intrusion Detection and Response Using Big Data Kleber Vieira, Fernando L. Koch , João Bosco M. Sobral, Carlos Becker Westphall, and Jorge Lopes de Souza Leão Abstract—We present a method for autonomic intrusion de- tection and response to optimize processes of cybersecurity in large distributed systems. These environments are characterized by technology fragmentation and complex operations making them highly susceptible to attacks like hijacking, man-in-the-middle, denial-of-service, phishing, and others. The autonomic intrusion response system introduces models of operational analysis and reaction based on the combination of autonomic computing and big data. We implemented a proof-of-concept and executed exper- iments that demonstrate significant improvement in effectiveness and scalability of the method in complex environments. Index Terms—Autonomic computing, big data, cybersecurity, distributed computing, intrusion detection systems. I. INTRODUCTION T HE widespread utilization of large distributed comput- ing solutions and growing number of cybersecurity is- sues [1] call for new methods to classify, understand, predict, and counter-react to impromptu cyberattacks. A report from PwC Consulting describes the impact of security breaches related to disruption of manufacturing, compromise of sensitive data, impact on shared services, damage of physical property, harm to human life, and others [2]. A recent report by the Brazilian Center for Studies, Response and Treatment of Security Incidents (CERT.br) shows an increase by 125.36% of distributed-denial- of-service (DDoS) attacks between the first quarters of 2015 and 2016 [3]. Cybersecurity threats are associated to illegitimate access to system processing and/or information in storage or during data transfer. The most common issues include hijacking, man-in- the-middle, denial-of-service, and phishing [4]. There are mul- tiple methods to detect and counter-react to intrusions such as quick detection of malicious or unauthorized actions [5], [6], and intelligent management methods of distribute computing [7], [8]. These methods usually work based on rule sets that once breached must be adjusted to contain the attack. As described Manuscript received January 22, 2019; revised June 2, 2019 and August 30, 2019; accepted September 21, 2019. The work of F. Koch was supported by the Brazilian CNPq Productivity in Technology and Innovation under Grant 307275/2015-9. (Corresponding author: Fernando L. Koch.) K. Vieira is with SENAI Institute of Embedded System, Brasil (e-mail: Klebermagno@gmail.com). F. L. Koch is with the IBM Services, NY, USA (e-mail: fkoch@acm.org). J. B. M. Sobral and C. B. Westphall are with the Universidade Federal de Santa Catarina, Florianopolis, SC 88040-900, Brazil (e-mail: bosco.sobral@ufsc.br; westphal@inf.ufsc.br). J. L. de S. Leão is with the Universidade Federal do Rio de Janeiro, Rio de Janeiro, RJ 21941-901, Brazil (e-mail: jorge.leao@ufrj.br). Digital Object Identifier 10.1109/JSYST.2019.2945555 in [9] and [10], if the adjustment process takes over 10 h, then an skilful intruder has 80% chance of success to carry on a widespread attack. Hence, there is a need for reaction strategies that provide online intrusion detection and fast response to prevent disruption, preserve security, and optimize operational costs. We introduce an strategy to combine autonomic computing and big data in processing large data volumes generated by system operations in near real-time. The autonomic intrusion response system (AIRS) introduces innovative models of op- erational analysis aiming to respond to attacks in milliseconds thus reducing the chance of success. We propose an approach based on the concept of self-healing, defined as the possibility to automated reaction to intrusion actions and its consequences. In our proposal, this concept has been implemented through methods to accumulate knowledge and improve the parameters of intrusion detection through the review of the parameters applied in the probability inference process. We applied the theory of expected utility for the rational choice of the rules to be applied during the detection process. The solution employs a configurable knowledge based that use regular expression. We experiment with signatures to detect attacks like hijacking, man-in-the-middle, denial-of-service, phishing, and others. This article contributes to the state-of-the art by the following: 1) providing a reference architecture describing the elements and interaction in the proposed system; 2) introducing a proof-of-concept implementation of a full- cycle interaction; 3) analyzing the performance when applied to real-world scenarios involving private and public cloud computing. In what follows, we elaborate on the background, state-of the-art, and technology gap. Section III outlines our proposal. Section IV presents the results from experimentation on private and public clouds. We conclude with a discussion of the results and future possibilities in Section V. II. BACKGROUND AND RELATED WORK There is a growing demand for Intrusion Detection Systems (IDS) to minimize the harms of hackers, crackers, and other cyber-criminals [11]–[13]. These systems encompass the fol- lowing statements: 1) Detection, usually performed automatically by monitoring patterns in the systems log to identify element behavior; 2) Warning, triggered by the detection of behavior deviance; 1937-9234 © 2019 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission. See http://www.ieee.org/publications_standards/publications/rights/index.html for more information.