A Novel Reinforcement Learning Framework for Online Adaptive Seizure Prediction Shouyi Wang, Wanpracha Art Chaovalitwongse Department of Industrial and Systems Engineering Rutgers, the State University of New Jersey Piscataway, NJ, USA shouyi@eden.rutgers.edu, wchaoval@rci.rutgers.edu Stephen Wong Robert Wood Johnson Medical School University of Medicine and Dentistry of New Jersey New Brunswick, NJ, USA wongst@umdnj.edu Abstract—Epileptic seizure prediction is still a very challenging and unsolved problem for medical professionals. The current bot- tleneck of seizure prediction techniques is the lack of flexibility for different patients with an incredible variety of epileptic seizures. This study proposes a novel self-adaptation mechanism which successfully combines reinforcement learning, online monitoring and adaptive control theory for seizure prediction. The proposed method eliminates a sophisticated threshold-tuning/optimization process, and has a great potential of flexibility and adaptability to a wide range of patients with various types of seizures. The proposed prediction system was tested on five patients with epilepsy. With the best parameter settings, it achieved an averaged accuracy of 71.34%, which is considerably better than a chance model. The autonomous adaptation property of the system offers a promising path towards development of practical online seizure prediction techniques for physicians and patients. Index Terms—biomedical data mining, adaptive seizure pre- diction, reinforcement learning, online monitoring I. I NTRODUCTION Epilepsy is one of the most common neurological disorders, affecting approximately 1% of the world’s population [6]. Epileptic seizures generally occur without any warning, and the shift between normal brain state and seizure onset is often considered as an abrupt phenomenon. The unpredictability of seizure occurrence represents a significant source of morbidity in patients with epilepsy. Patients with epilepsy frequently suffer from seizure-related injuries due to loss of motor con- trol, loss of consciousness or delayed reactivity during seizure onset [16]. At the moment, no technology is available to provide a warning to these patients prior to seizure onset. The ability to predict the occurrence of impending seizures could significantly improve the quality of life of epileptic patients. Seizure prediction may also lead to novel therapeutic strategies for seizure control. For example, a prediction-triggered closed- loop treatment may replace the traditional method of taking anticonvulsant drugs daily. Such temporally targeted therapy methods may largely reduce the side effects of current chronic drug treatments as reported in [3]. Perhaps most importantly, seizure prediction could give patients with epilepsy a greater sense of control over their lives. One crucial question in seizure prediction is that whether an identifiable pre-seizure state exists. Over the recent years, there has been accumulating evidence indicating that a transitional pre-seizure state does exist prior to seizure onset. The majority of the quantitative evidence supporting the existence of a pre- seizure state is derived from Electroencephalography (EEG) analyses of epileptic seizures. For example, Lehnertz and Elger [12] showed that the correlation dimension decreases prior to seizures. Le van Quyen et al. [19] reported a reduction in the dynamical similarity index before seizure occurrence. Iasemidis et al. [8] noted premonitory pre-seizure changes based on the analysis of dynamical entrainment. Mormann et al. [15] observed a pre-seizure drop in phase synchronization up to hours prior to seizure onset. Recent studies suggests that four stages are evolved in a seizure process: normal, pre- seizure, seizure onset and post-seizure [13]. In the mid-1970s, Viglione and Walsh started the first pioneering project to investigate the predictability of seizures based on EEG data [25]. Since then, many studies have been carried out aiming to predict epileptic seizures based on EEG data. An extensive survey of EEG-based seizure prediction techniques can be found in [14]. In general, most of current seizure prediction methods mainly have two steps. Firstly, EEG features are extracted from a sliding moving window. Then each windowed EEG epoch is classified as either pre- seizure or normal based on a threshold level. Whenever a windowed EEG epoch is classified as pre-seizure, a warning alarm is triggered indicating that an impending seizure may occur within a pre-defined prediction horizon. Although these methods have shown good results for some patients, the reliability and repeatability of the results have been questioned when they were tested on other EEG datasets. Many of the earlier optimistic findings cannot be reproduced or achieved poor performance in extended EEG datasets in later studies as reported in [2]. This is not surprising since the optimal threshold obtained from a few number of patients may not be suitable to many others. The current seizure prediction techniques are still in their early stage. Before considering clinical applications, the evaluation of a seizure prediction method is still to test whether the prediction performance is consistently better than a chance level in most research efforts [27], [1], [11]. The biggest challenge of seizure prediction is the high inter- and intra-individual variability among the patients with epilepsy. The high variability makes the traditional nonadap- 2010 IEEE International Conference on Bioinformatics and Biomedicine 978-1-4244-8305-1/10/$26.00 ©2010 IEEE 499