Layered Security Architecture for Threat Management using Multi-Agent System Vandana Gandotra, Archana Singhal, Punam Bedi Department of Computer Science University of Delhi, Delhi-110007 India +91-11-27667591 vandanarla@yahoo.co.in, singhal_archana@yahoo.com, pbedi@acm.org ABSTRACT The increasing complexity of software systems along with expanding connectivity has necessitated the evolution of an integrated security framework adopting innovative techniques for secure software systems. This paper proposes a layered security architecture for threat management using a multi-agent system to meet the above objective. Layer- 1 of this framework is designed for elicitation of realistic and flawless security requirements. Layer-2 uses a Multi-Agent system planning for avoidance of threats optimally. In this mechanism autonomous agents interact and coordinate with each other to achieve the common goal of software security. An adaptive defense mechanism using Meta-Agents in multi-agent system in conjunction with fuzzy logic to counter the adaptive and compound threats is the responsibility of Layer-3. Guidelines proposed in this paper have augmented this security architecture as a two-fold defensive strategy to ensure that a hacker is not able to tamper data even if they penetrate the periphery defenses. These proactive steps can be implemented during the design and development phases of the software life cycle in an incremental way as per the budget and security requirements of a software project. A case study on internet banking is included in the paper to describe the proposed security framework. Categories and Subject Descriptors D.2.9 [Software Engineering]: Management- Life Cycle. I.2.11 [Artificial Intelligence]: Distributed Artificial Intelligence- Multi- Agent System. General Terms Management, Design, Security. Keywords Layered security architecture, Hybrid technique, Fuzzy logic, Meta- agents, Multi-agent system planning, Adaptive defense mechanism. 1. INTRODUCTION There has been a steep rise in the number of security breach incidents in last few decades due to complex software systems [5] [15]. Unfortunately, it is not possible to prevent threats with the existing security measures being adopted in isolation in the present day global security scenario [2]. This has led us to propose a new proactive approach that can address this problem in a comprehensive manner and has been evolved by integrating various paradigms to match the evolutionary nature of threats manifested these days. In this paper, we present a layered security architecture as a multi-faceted security mechanism to augment and complement traditional threat management. The proposed framework overcomes the deficiency of shallowness in a single-layer defense strategy and provides multi-level security cover to avert threats [8]. Hybrid Technique, [7] which has been evolved by overlapping the strengths of misuse cases and attack trees, has been adopted at Layer-1 for elicitation of realistic and meaningful security requirements. In this technique threats are represented using a Hybrid Process Diagram that helps the system designers to receive as much information as possible for design and development of secure software systems. Since the ‘Adaptivity and Intelligence’ is the most important attribute of a security mechanism desired these days to meet new threat perceptions, multi-agent system planning has been adopted in Layer-2 to meet new security requirements. In Multi Agent System Planning for Threat Avoidance (MASPTA) [1], agents work in unison to avoid threats to a web based system by executing their predetermined action plans as per the schedule generated. In the present paper, MASPTA has been enhanced to achieve the objective of threat avoidance optimally by generating the most promising remedial plan. This has been made possible by using Single Participant Multiple Attribute (SPMA) and Technique for Order Preference by Similarity to Ideal Solution (TOPSIS) multiple attribute decision making methods. Here, agents are inducted only in the attack paths of the selected remedial plan to prevent the attackers from compromising the security of the system optimally. In the above mechanism, agents succeed in averting the threat keeping the system in safe mode. However in case they fail, it will lead to catastrophic failure which is unacceptable in this environment. This jinx of brittleness has been broken using Fuzzy Logic at Layer-3 by generating an intermediate stage i.e. Partially Secure State (Yellow Zone) between the Safe State (Green Zone) and the Failed State (Red Zone). This has helped in averting the system from reaching the Red Zone consequent to introduction of Yellow Zone [9]. In this work, the above concept has been adopted in the proposed adaptive defense mechanism in Layer-3 to meet the innovative and sophisticated threats from present day hackers. This enhancement has been made possible by inducting Meta-Agents in multi-agent system in conjunction with Fuzzy Logic as shown in Figure 4. Meta- Agents which are adaptive and reactive in nature are used as management, coordination, matching and checking agents [16] [17]. In this paper, a monitoring meta-agent has been introduced to monitor the performance of active agents employed in Layer-2 and also to monitor the variations in the security goals. The other meta-agents in this framework act autonomously, reason and take decisions to save the system from being compromised in the light of variations in the security level generated as output of Fuzzy Inference System. Although the above framework provides layered protection against present day threats, hackers these days are intelligent enough to peep into our defenses using innovative and adaptive threats to intrude into the system. It is therefore essential that even if the hacker penetrates the periphery defenses and enters the system, he should not be able to compromise or tamper data. Some strategic steps in this direction have also been proposed in this paper to strengthen in-built security mechanism of this framework to save the system from malicious users in such cases. Thus adoption of the proposed proactive steps in threat management detailed above provides necessary guidelines to the software developers to enhance and augment the security of software systems. This framework can be judiciously incorporated at the design and development stage of SDLC to achieve this aim. The developers can use one or more of these proactive steps as per the security requirements / security policy of the system and the type of data being handled ACM SIGSOFT Software Engineering Notes Page 1 September 2011 Volume 36 Number 5 DOI: 10.1145/2020976.2020984 http://doi.acm.org/10.1145/2020976.2020984