Counteracting security attacks in virtual machines in the cloud using property based attestation Vijay Varadharajan n , Udaya Tupakula Advanced Cyber Security Research Centre, Faculty of Science, Macquarie University, Sydney, Australia article info Article history: Received 7 January 2013 Received in revised form 11 July 2013 Accepted 5 August 2013 Available online 17 August 2013 Keywords: Trusted computing TPM attestation Cloud Virtual machine monitors Rootkits Zero day attacks Malware abstract Cloud computing technologies are receiving a great deal of attention. Furthermore most of the hardware devices such as the PCs and mobile phones are increasingly having a trusted component called Trusted Platform Module embedded in them, which helps to measure the state of the platform and hence reason about its trust. Recently attestation techniques such as binary attestation and property based attestation techniques have been proposed based on the TPM. In this paper, we propose a novel trust enhanced security model for cloud services that helps to detect and prevent security attacks in cloud infra- structures using trusted attestation techniques. We consider a cloud architecture where different services are hosted on virtualized systems on the cloud by multiple cloud customers (multi-tenants). We consider attacker model and various attack scenarios for such hosted services in the cloud. Our trust enhanced security model enables the cloud service provider to certify certain security properties of the tenant virtual machines and services running on them. These properties are then used to detect and minimise attacks between the cloud tenants running virtual machines on the infrastructure and its customers as well as increase the assurance of the tenant virtual machine transactions. If there is a variation in the behaviour of the tenant virtual machine from the certified properties, the model allows us to dynamically isolate the tenant virtual machine or even terminate the malicious services on a fine granular basis. The paper describes the design and implementation of the proposed model and discusses how it deals with the different attack scenarios. We also show that our model is beneficial for the cloud service providers, cloud customers running tenant virtual machines as well as the customers using the services provided by these tenant virtual machines. & 2013 Elsevier Ltd. All rights reserved. 1. Introduction Distributed systems have fundamentally changed the way individuals and enterprises share, process and store information today. Security issues play a vital role in distributed systems, as greater availability and access to information in turn imply that there is a greater need to protect them. To address these issues, several security techniques, mechanisms and systems have been proposed over the years (Ferraiolo and Kuhn, 1992; Jajodia et al., 1997; DeTreville, 2002; Li and Mitchell, 2003; Herzberg et al., 2000; Blaze et al., 1996, 1999). Many of these systems have addressed the authentication and authorisation requirements that relate to human users. They make some basic assumptions about the state of the platform that is hosting and running the systems software and applications. There is an inherent trust that is placed on the underlying platform when a higher level application or user is authenticated or authorised. In the current networked world with heterogeneous platforms and numerous software applications and system software running on these platforms, it is important that such underlying trust assumption about the system state be properly examined. There are several reasons for this. First, computing platforms have become very powerful and can run many applications simulta- neously. In particular, as the number of software applications increases, greater is the potential for security vulnerabilities to arise. These vulnerabilities in turn make the platform more vulnerable to attacks. Second, attacks themselves are becoming more and more sophisticated. Furthermore, attackers also have easier access to ready-made tools that enable exploitation of platform vulnerabilities more effective. Third, platforms are being shared by multiple users and applications (belonging to different users) both simultaneously as well as at different times. Therefore there is a great chance of the platform being left in a vulnerable state as different users and applications run. Finally, because platforms have become much more complex today, users them- selves are unaware of their platform vulnerabilities. Let us assume, for example, a user is authorised to download and execute a confidential file on his/her platform. A typical authentication and Contents lists available at ScienceDirect journal homepage: www.elsevier.com/locate/jnca Journal of Network and Computer Applications 1084-8045/$ - see front matter & 2013 Elsevier Ltd. All rights reserved. http://dx.doi.org/10.1016/j.jnca.2013.08.002 n Corresponding author. Tel.: þ61 2 98509534. E-mail addresses: vijay.varadharajan@mq.edu.au (V. Varadharajan). Journal of Network and Computer Applications 40 (2014) 31–45