0018-9294 (c) 2018 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission. See http://www.ieee.org/publications_standards/publications/rights/index.html for more information. This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI 10.1109/TBME.2018.2875024, IEEE Transactions on Biomedical Engineering TBME-00573-2018.R2 1 Abstract— The performance of an existing Devanagari Script (DS) input based P300 speller with conventional machine learning techniques suffers from low information transfer rate (ITR). This occurs due to its required large size of display i.e. 8 x 8 row-column (RC) paradigm which exhibits issues like crowding effect, adjacency, fatigue, task difficulty and required large number of trials for character recognition. For P300 detection, deep learning algorithms have shown the state of art performance compared to the conventional machine learning algorithms in the recent past. Therefore, authors have motivated to develop a deep learning architecture for DS based P300 speller which can detect the target characters more accurately and in less number of trials. For this, two proven deep learning algorithms, stacked autoencoder (SAE) and deep convolution neural network (DCNN) have been adopted. For further bettering their performances, batch normalization and innovative double batch training included here to achieve accelerated training and alleviate the problem of overfitting. Additionally, a leaky ReLU activation function has also been used in DCNN to overcome dying ReLU problem. The experiments have been performed on self-generated dataset of 20 Devanagari words with 79 characters acquired from 10 subjects using 16 channel actiCAP Xpress EEG recorder. The experimental results illustrated that the proposed DCNN is able to detect 88.22 % correct targets in just 3 trials. Moreover, it also provides ITR of 20.58 bits per minutes which is significantly higher than existing techniques. Index Terms – Brain-computer interface (BCI), Devanagari script (DS), Stacked Autoencoder (SAE), Deep convolution neural network (DCNN), P300 speller I. INTRODUCTION Patients who are suffering from chronic neuromuscular disorders but having some cognitive abilities [1] requires a medium which allows them to communicate with the outer world. A P300-based brain-computer interface (BCI) provides such communication medium which utilizes their brain signals (EEG signals) to interact with the machine without involving any muscular movements [2], [3], [4]. P300, the neuro-cognitive response of the brain, elicits after external visual stimulation based on the principle of the oddball paradigm. This event-related potential (ERP) has a positive deflection between 250-500ms [5] with its peak approximately at 300ms [2], as shown in Fig. 1. However, it’s amplitude and the latency may vary based on the psycho-physiological condition of the patients, as well as the spatiotemporal arrangement of the stimuli [5]. Fig. 1. EEG/ERP response of target and non-target P300 component. In late 80’s, a very first P300 speller using a 6 × 6 matrix of English letters and other characters was developed to spell English words [6]. It is always convenient for subjects to communicate in their native languages. Hence, development of P300 speller systems based in native scripts is expected. In view of that many research groups have taken interest in developing P300 speller display paradigms in native languages other than English in the recent time. P300 speller for Chinese [7], Japanese [8], Arabic [9] and very recently by our group Devanagari script-based spellers have been developed [2], [10], [11]. In previous studies on DS based P300 speller [2], [10], [11], [12], row-column (RC) paradigm with 8 x 8 matrix of Devanagari letters and other characters were implemented shown in Fig. 2. Fig. 2. The 8 x 8 matrix-based Devanagari display paradigm for P300 speller. For the detection of the characters in existing DS based p300 speller, most successful Support Vector Machine (SVM) [10] and followed by its weighted ensemble variant [33] have been used to further reduce the false negative rate (FNR). However, though performance of DS based P300 speller have enhanced and shown highest accuracy of 94.2% but with 15 number of trials [10]. Further, it has been also reported that with reduced number of trails, the accuracy reduces even after adapting the Improving Performance of Devanagari Script Input-Based P300 Speller Using Deep Learning G. B. Kshirsagar, Student Member, IEEE, and N. D. Londhe, Senior Member, IEEE A positive peak after flashing of the target character