Hindawi Publishing Corporation EURASIP Journal on Wireless Communications and Networking Volume 2011, Article ID 210746, 14 pages doi:10.1155/2011/210746 Research Article A Novel Approach to Detect Network Attacks Using G-HMM-Based Temporal Relations between Internet Protocol Packets Taeshik Shon, 1 Kyusuk Han, 2 James J. (Jong Hyuk) Park, 3 and Hangbae Chang 4 1 Division of Information and Computer Engineering, College of Information Technology, Ajou University, Suwon 443-749, Republic of Korea 2 Department of Information and Communication Engineering, Korea Advanced Institute of Science and Technology, 119 Munjiro, Yuseong-gu, Daejeon 305-701, Republic of Korea 3 Department of Computer Science and Engineering, Seoul National University of Science and Technology, 172 Gongneung 2-Dong, Nowon, Seoul 139-743, Republic of Korea 4 Department of Business Administration, Daejin University, San 11-1, Sundan-Dong, Pocheon-Si, Gyunggi-Do 487-711, Republic of Korea Correspondence should be addressed to Hangbae Chang, hbchang@daejin.ac.kr Received 20 August 2010; Accepted 19 January 2011 Academic Editor: Binod Vaidya Copyright © 2011 Taeshik Shon et al. This is an open access article distributed under the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited. This paper introduces novel attack detection approaches on mobile and wireless device security and network which consider temporal relations between internet packets. In this paper we first present a field selection technique using a Genetic Algorithm and generate a Packet-based Mining Association Rule from an original Mining Association Rule for Support Vector Machine in mobile and wireless network environment. Through the preprocessing with PMAR, SVM inputs can account for time variation between packets in mobile and wireless network. Third, we present Gaussian observation Hidden Markov Model to exploit the hidden relationships between packets based on probabilistic estimation. In our G-HMM approach, we also apply G-HMM feature reduction for better initialization. We demonstrate the usefulness of our SVM and G-HMM approaches with GA on MIT Lincoln Lab datasets and a live dataset that we captured on a real mobile and wireless network. Moreover, experimental results are verified by m-fold cross-validation test. 1. Introduction The world-wide connectivity and the growing importance of internet have greatly increased the potential damage, which is inflicted by attacks over the internet. One of the conventional methods for detecting such attacks uses attack signatures that reside in the attacking program. The method requires human management to find and analyze attacks, make rules, and deploy the rules. The most serious disadvantage of these signature schemes is that it is difficult to detect the unknown and new attacks. Anomaly detection algorithms use a normal behavior model for detecting unexpected behaviors as measures. Many anomaly detection methods have been researched in order to solve the signature schemes problem by using machine learning algorithms. There are two categories of machine learning for detecting anomalies; supervised methods make use of preexisting knowledge and unsupervised methods do not. Several efforts to design anomaly detection algorithms using supervised methods are described in [1–5]. The researches of Anderson at SRI [1, 2] and Cabrera et al. [3] deal with statistical methods for intrusion detection. Lee and Xiang’s research [4] is about theoretical measures for anomaly detection, and Ryan [5] uses artificial neural networks with supervised learning. In contrast, unsupervised schemes make appropriate labels for a given dataset automatically. Anomaly detection methods with unsupervised features are explained in [6–10]. MINDS [6] is based on data mining and data clustering methods. The researches of Eskin et al. [7] and Portnoy et al. [8] were used to detect anomaly attacks without preexisting knowledge.