1556-6013 (c) 2018 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission. See http://www.ieee.org/publications_standards/publications/rights/index.html for more information. This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI 10.1109/TIFS.2018.2886465, IEEE Transactions on Information Forensics and Security IEEE TRANSACTIONS ON INFORMATION FORENSICS AND SECURITY 1 A Two-Step Approach To Optimal Selection Of Alerts For Investigation in a CSOC Ankit Shah, Rajesh Ganesan, Sushil Jajodia, Fellow, IEEE and Hasan Cam, Senior Member, IEEE Abstract—A Cyber Security Operations Center (CSOC) is responsible for investigating all the alerts generated from the intrusion detection systems (IDSs) to identify suspicious activities in a timely manner. There exists a critical gap between the time needed (demand) and the time available (limited analyst resource) for alert investigation at a CSOC. Hence, alert prioritization is important, for which CSOCs employ ad-hoc filtering methods to prune and triage the alerts that are presented to the analysts for investigation. One of the major drawbacks of the ad-hoc methods is that they do not comprehensively take into consideration the organization-specific factors such as mission and asset criticality, CSOC resource availability, demand variations, and the desired CSOC performance metrics. Hence, an ad-hoc triaging (or pri- oritization) method is insufficient, and an intelligent method for optimal selection of alerts that considers the above organization- specific factors must be developed, which is described as a two- step process in this research. First, a composite risk-score of each alert is determined using a Quantitative Value Function (QVF) hierarchy process, which takes into account several organization- specific factors. Second, an optimization model selects a list of alerts for investigation that optimizes the CSOC performance metrics for a given demand subject to its resource constraints. Experimental results show that the alerts that pertain to mission criticalities are handled in a timelier manner as compared to current practices at the CSOCs. The average persistence time of an alert in the CSOC system is also shown to significantly reduce with this new approach, which is a paradigm shift in providing a stronger cyber-defense system by protecting the critical constituents of an organization. Index Terms—Alert Prioritization, CSOC, Optimization, Quantitative Value Function Hierarchy Model, and Risk-score I. I NTRODUCTION A Cyber Security Operations Center (CSOC) is responsible for the timely identification of suspicious activities in an organization’s network and thoroughly investigating them to prevent (or recover from) the cyber incidents. A CSOC com- prises of cyber analysts who continuously monitor interesting activities, in the form of alerts, generated from the intrusion detection systems (IDSs). Analysts are expected to thoroughly investigate all the alerts and classify each alert as suspicious, or innocuous through a triaging process. Suspicious alerts are given a closer inspection and a portion of them are A. Shah, R. Ganesan and S. Jajodia are with the Center for Secure Information Systems, George Mason University, Fairfax, VA 20030 USA e- mail: {ashah20, rganesan, jajodia}@gmu.edu. H. Cam is with the U.S. Army Research Laboratory, Adelphi, MD 20783 USA e-mail: hasan.cam.civ@mail.mil. Shah, Ganesan, and Jajodia were partially supported by the Army Research Office under grants W911NF-13-1-0421 and W911NF-15-1-0576 and by the Office of Naval Research under grant N00014-15-1-2007. Manuscript received xxxx 1xx, 2018; revised xxxx xx, 2018. Fig. 1. Alert Data Hierarchy for a CSOC categorized as significant alerts (incidents or events) according to the categories shown in [1] and are escalated for further investigation. Figure 1 provides a visual representation of a typical alert data hierarchy for a CSOC organization [2]. Figure 2 shows the alert analysis process at a CSOC. The paper focuses on the triaging and selection to form alert queue block of the figure, which is highlighted. In an ideal scenario, a sufficient number of analysts are available at a CSOC that are required for a timely investigation of all the alerts generated. However, in reality, due to reasons such as (a) not having enough analysts because of budgetary or availability constraints, or (b) an unexpected increase in alert generation rates, the time required to investigate all the alerts (demand) often exceeds the time available to investigate them (supply i.e. available analyst resource). Such a disparity between demand for and supply of investigation time results in an incomplete or inaccurate analysis, consequently, a failure to detect malicious activities [3], which increases the security threat to the critical assets of an organization. The above motivates the importance of alert prioritization (triaging) and selection of the suspicious alerts generated by the triaging process. It is a common practice at CSOCs to prune the alerts that are presented to the analysts for investigation by using ad- hoc filtering methods such as First-In-First-Out (FIFO) or priority queueing based on alert severity (PQ). These methods are described in the related literature section of the paper. Unfortunately, these ad-hoc methods do not take into account a holistic view of the security of an organization [4], and often miss out on the critical alerts because they ignore the combined effect of the following factors or alert characteristics during triaging such as (a) the severity of the alert as determined during triaging (SNORT priority ranking, or expert knowledge elicitation [5]), (b) the organization-specific characteristics as-