Dynamic Monitoring of Malicious Activity in Software Systems M. Debbabi , M. Girard , L. Poulin , M. Salois , N. Tawbi LSFM Group Department of Computer Science Laval University Qu´ ebec, Canada http://www.ulaval.ca/lsfm Defence Research Establishment Valcartier 2459, Pie-XI blvd North Val-B´ elair, Qu´ ebec, Canada G3J 1X5 http://www.drev.dnd.ca October 2000 Abstract Because of time and budget constraints, organisations are turning more and more to Commercial-Off-The-Shelf (COTS) software rather than developing in-house soft- ware. This situation gives rise to great concerns over safety, security, and reliability in critical information sys- tems. This paper presents a research effort to help manage the risk associated with COTS integration through the ex- ploitation of a dynamic monitor. The strategy consists of inserting appropriate drivers to control accesses to critical resources: files, communication ports, the registry, and the creation/destruction of processes and threads. A prototype monitor called DaMon has been designed and developed to run in Windows NT on an Intel box, to demonstrate the feasibility of reactive monitoring to meet a formal security specification. 1 Introduction A set of tools to circumscribe harmful effects and even to eliminate them would be a significant asset in an environment requiring tight control of security. With the increasingly high use of Commercial-Off-The-Shelf (COTS) software, it is imperative for many organisations to assure themselves that any out-sourced product will meet strict standards of safety, security, and reliability. One has no guarantees that a piece of software does not contain malicious code or that it does only what it is sup- posed to do. For example, an application could be sending private information without consent. Even if the program specifications do not include sending information exter- nally, proof that it does not do so is hard to come by. Even in this simple example, it appears obvious that some form of proof of good conduct is required from the vendor. Barring that, other tools are needed to check and control this kind of malicious behaviour in already in- stalled software. At that time, of course, only the exe- cutable code and its accompanying compiled libraries are This research is funded by a research contract from the Defence Re- search Establisment Valcartier (DREV), Qu´ ebec, Canada. available, which further complicates matters. To deal with this problem, in 1997 the Defence Re- search Establishment Valcartier (DREV) contracted a study by the Language, Semantics, and Formal Method (LSFM) group at Laval University. The present Mali- COTS project team consists of about 12 graduate students, four professors, and two DREV scientists. The platform chosen for the project is Windows NT running on an Intel box. Preliminary studies identified three main areas for in- vestigation: 1. Certifying compilers 2. Static analysis of executables 3. Dynamic analysis of executables The first area is the most promising in the long term since it works on source code, which provides the great- est flexibility. This approach tries to extract enough infor- mation from the source code to allow formal verification, without compromising the intellectual property rights ap- plicable to the code [2]. The second approach is severely limited since one does not usually have access to COTS source code, and disas- sembling such software poses legal problems. Nonethe- less, some investigations are being made by the team to adapt static analysis techniques to assembly-language source code. The third set of techniques has three possible facets. One is to build a kind of virtual machine that will check ev- ery instruction executed by the processor for validity [11]. This approach has the same limitations as static analysis since it works at assembly level. It has another limitation: runtime overhead. A second facet is to do log analysis. This tactic has two major flaws: (1) it is highly limited by what the system can log, although it is possible to extend that capability; (2) it is only reactive, since one cannot log what has not yet happened, thus it is rather limited in its ability to prevent some undesirable action from occur- ring. The third facet is to build a tool to monitor critical resources for malicious actions. In this case, a security 1