A Heuristic Approach for Firewall Policy Optimization El-Sayed M. El-Alfy College of Computer Sciences and Engineering King Fahd University of Petroleum and Minerals Dhahran 31261, Saudi Arabia alfy@kfupm.edu.sa Abstract ⎯ A primary goal of this paper is to develop a heuristic approach based on genetic algorithms to enhance the firewall performance. Typical firewall policies may have thousands of rules and determining an optimal rule order that minimizes the average number of rule comparisons while maintaining the policy integrity is proven to be NP-hard. This problem is formulated as a binary integer program for which an optimal solution is obtained using the branch-and-bound technique. Then an alternative solution approach is devised based on genetic algorithms. Several experiments are conducted to evaluate the effectiveness of the proposed approach as compared to other rule-ordering techniques. Empirical results show the potential and flexibility of the proposed approach. Keywords ⎯ Network security, firewalls, access control, and genetic algorithms. 1. Introduction Providing solutions for network security has recently attracted considerable attention of several network researchers and practitioners. Securing private transactions over the Internet is gaining importance everyday and several mechanisms and standards have been developed to protect message confidentiality, preserve message integrity, authenticate the sender, and ensure non-repudiation. However, these security mechanisms do not prevent an intruder from sending a harmful message to an enterprise network connected to the Internet. Firewalls [1, 2] are commonly used solutions as security barriers to stop such attacks and significantly increase the security (if they are properly configured). A firewall is usually installed between an organization’s private network and the Internet to control and prevent unauthorized access to the network resources. Firewalls come in many different forms but the primary activity of a firewall is filtering packets that pass through it between the protected network and the Internet. The filtering operation is essentially based on an ordered set of rules. Typical firewalls may have thousands of rules. Each rule specifies whether to drop or pass a received packet based on the information contained in the packet headers. In list-based policy representation, when a packet arrives at the firewall, it is sequentially compared against rules in the policy list until a match (typically first match) is found or reaching the end of the list. To make the policy complete, i.e. always finding a match for every possible legal packet, a deny-all rule is often inserted at the end of the list [9]. Once a match is found, the corresponding action is applied to drop or pass the packet. The computational complexity of the filtering operation depends on the depth of finding a matched rule in the rule list. A good order of rules is one that reduces the number of comparisons. When the new policy always gives the same action for each incoming packet, it is functionally equivalent to the original policy. However, since rules are not necessarily mutually exclusive (i.e. a packet can match more than one rule), improper ordering can cause security and performance problems. Hence, it is required to reorder the rules in the policy list to minimize the average number of comparisons while maintaining the precedence relationship between intersecting rules to preserve the integrity of the original policy. The optimal rule ordering with precedence constraints is among the hardest combinatorial optimization problems for which no efficient solution algorithm is known yet to run in polynomial time. Since this problem is equivalent to scheduling jobs with precedence constraints for a single processor, it has been shown to be NP-hard [3]. Thus, a straightforward sorting is not possible. A simple heuristic along with a sorting algorithm is presented in [3]. In this method, rules are arranged in non-increasing order of matching probabilities unless precedence relationships exist between neighboring rules. Although this method can improve the performance of a list-oriented firewall, it is possible that one rule can prevent other rules from being reordered. In addition, the degree of improvement depends on the linear arrangement generated by the topological sort algorithm. Recently, genetic algorithms have been applied to a number of engineering design problems with promising results [4]. As meta-heuristic search techniques, they provide flexibility in global optimization and do not require problem-specific knowledge in order to get good solutions. In this paper, we apply a heuristic approach based on genetic algorithms to optimize the firewall security policy. The goal is to find a rule order that minimizes the average number of comparisons while maintaining the precedence relationships between rules. To evaluate the effectiveness of the proposed solution approach, the results are compared with two other firewall optimization methods. The first method is an exact optimal solution approach for binary integer program namely the branch-and-bound (BB) method. The second method is ISBN 978-89-5519-131-8 93560 - 1782 - Feb. 12-14, 2007 ICACT2007