Detection of Worm Propagation Engines in the System Call Domain using Colored Petri Nets A. Tokhtabaev, V. Skormin, Center for Advanced Information Technologies, Binghamton University E-mail: {atokhta1, vskormin}@binghmaton.edu While network worms are different in payloads and utilized exploits, they all have one common component, the propagation engine. It is important that the number of conceptually distinctive propagation engines employed by existing network worms is quite limited. This paper presents a novel approach for detecting attacks perpetrated by network worms. It implies the recognition of the propagation engine functionality in the process behavior on the system call level and attributing it to the shell code activity being the first stage in the worm proliferation. We suggest an elegant way to trace the behavior of a computer process in the system call domain and assess its current functionality through the utilization of Colored Petri Nets. We developed, tested and evaluated a Propagation Engine Detector (PED) system which detects the worm shell code activity performed by a process during the attack. Moreover, PED recognizes the type of propagation engine employed by the attacking worm. Keywords: worm propagation engine, IDS, Colored Petri Nets, System calls, network worms 1. Introduction Our ever-growing dependence on Internet is accompanied by ever-growing concerns about the networks vulnerability to information attacks and the dependability of the existing network security systems. Major threats, well recognized by government, private institutions and individual users, are stemming primarily from self-proliferating malicious software such as network worms. Network worms perpetrating remote code execution attack, such as buffer overflow, stack overflow, heap overflow, etc., have two vital components, propagation engine (shell code) and exploit. The shell code being a necessary part of the propagation engine is executed by the vulnerable process just after the exploit vector rerouted the control flow. The shell code creates specific conditions which are utilized by the attacking worm to complete the propagation session. Hence, every network worm performing remote code execution attack, employs a particular type of propagation engine and corresponding shell code in every attack to replicate itself into the victim machine. The adversaries usually utilize standard propagation engines along with available exploits of the selected vulnerability. This could be explained by the fact that reverse- engineering of the services, determining vulnerability, developing the exploit and producing specific shell code (propagation engine) requires special experience and knowledge which is possessed by a small community of computer professional. The largest part of the worms is written by so-called “script–kiddies” [1], who utilize available exploits with standard shell