XACML Policy Performance Evaluation Using a Flexible Load Testing Framework Bernard Butler, Brendan Jennings, Dmitri Botvich FAME Telecommunications Software & Systems Group Waterford Institute of Technology Ireland {bbutler,bjennings,dbotvich}@tssg.org ABSTRACT The performance and scalability of access control systems is growing more important as organisations deploy ever more complex communications and content management systems. Fine-grained access control is becoming more pervasive, so decisions are more frequent and policy sets are larger. We outline a flexible performance testing framework that ac- cepts XACML PDP implementations (in the server com- ponent) and submits representative access control requests (from the client component) in a representative temporal or- dering. The framework includes instrumentation and anal- ysis modules to support performance experiments. We de- scribe an initial realization of the framework and report on initial experiments comparing the performance of the SunX- ACML and Enterprise XACML PDPs. Categories and Subject Descriptors D.2.8 [SOFTWARE ENGINEERING]: Metrics—Per- formance measures ; D.4.6 [OPERATING SYSTEMS]: Security and protection—Access controls, Information flow controls General Terms Security, Performance, Measurement Keywords Access control policies, performance evaluation, measure- ment testbed 1. INTRODUCTION Policy Decision Point (PDP) performance is an important access control system requirement. In larger organisations, access decisions depend on the context of an access request, so fine-grained access control is needed to implement se- curity policies with complex boundaries between permitted and denied behaviour. There are more access requests, hence policy evaluations and each policy evaluation takes longer as policy sets grow larger. As an example, policy control of instant messaging com- munications in enterprises causes large numbers of policy evaluations, particularly in group-chat scenarios, where the Copyright is held by the author/owner(s). CCS’10, October 4–8, 2010, Chicago, Illinois, USA. ACM 978-1-4503-0244-9/10/10. access control system must decide which participant pairs can communicate. Such policy control is needed in organi- sations where Chinese Walls [1] must be maintained between groups for regulatory reasons. Many enterprise-level access control systems encode access controls as XACML (the eXtensible Access Control mod- elling Language) [7] hence researchers focus on XACML poli- cies and requests and their use in PEPs (Policy Execution Points) and XACML-based PDPs. It is relatively easy to scale out the (stateless) PEP func- tion, but not the (stateful) PDP function. Typical perfor- mance measures of a PDP set include latency and through- put, so any testbed needs to compute both. Some researchers advocate “black box” approaches such as caching frequently encountered request-result pairs. Alternatively, given one or more one of the policy set, request profiles or PDP source code, “white-box” approaches are possible. XACML poli- cies can be improved by categorisation, reordering and clus- tering [4], numericalisation and simplification to tree struc- tures [3], etc. XACML policies can also be replaced with an equivalent Description Logic formulation [2]. Generally the evidence presented by researchers is based on comparisons with the Sun XACML reference implemen- tation [8] often using unpublished policies and requests. Hence it is difficult to compare one approach with another, or to determine what tradeoffs occur. We propose a performance testbed for access control implementations to facilitate re- search into the performance and scalability problems facing XACML-based access control. The aim of our work is to provide a flexible (easily configured) framework, enabling researchers to perform quantitative experiments under rep- resentative, controlled and repeatable conditions. 2. RELATED WORK The problem of generating a large and representative set of policy requests for performance evaluation is related to that of generating a test set that covers as many of the pol- icy conditions as possible. By ensuring full coverage, all policy conditions are checked and so there is a path to each terminal node in the decision tree inferred from the policy set [5]. [5] also describes how Margrave can be used to de- termine redundant rules in a complex policy set. [6] describe how policy mutation testing may be used to determine how well a given test set of XACML requests discovers faults (deliberately injected as mutations ) in policy sets. Data clustering has been applied to characterise policies and hence improve PDP performance [4].