2576 IEEE TRANSACTIONS ON BIOMEDICAL ENGINEERING, VOL. 59, NO. 9, SEPTEMBER 2012 Spike Detection and Clustering With Unsupervised Wavelet Optimization in Extracellular Neural Recordings Vahid Shalchyan, Student Member, IEEE, Winnie Jensen, Member, IEEE, and Dario Farina*, Senior Member, IEEE Abstract—Automatic and accurate detection of action potentials of unknown waveforms in noisy extracellular neural recordings is an important requirement for developing brain–computer inter- faces. This study introduces a new, wavelet-based manifestation variable that combines the wavelet shrinkage denoising with mul- tiscale edge detection for robustly detecting and finding the occur- rence time of action potentials in noisy signals. To further improve the detection performance by eliminating the dependence of the method to the choice of the mother wavelet, we propose an unsu- pervised optimization for best basis selection. Moreover, another unsupervised criterion based on a correlation similarity measure was defined to update the wavelet selection during the clustering to improve the spike sorting performance. The proposed method was compared to several previously proposed methods by using a wide range of realistic simulated data as well as selected experi- mental recordings of intracortical signals from freely moving rats. The detection performance of the proposed method substantially surpassed previous methods for all signals tested. Moreover, up- dating the wavelet selection for the clustering task was shown to improve the classification performance with respect to maintaining the same wavelet as for the detection stage. Index Terms—Action potential (APs), extracellular recording, spike detection, spike sorting, unsupervised optimization, wavelet design. I. INTRODUCTION E XTRACELLULAR recordings from neuronal activities of the brain can be used as a source of information for brain– Manuscript received November 28, 2011; revised February 22, 2012 and May 29, 2012; accepted June 8, 2012. Date of publication July 3, 2012; date of current version August 16, 2012. This work was supported in part by a doctoral scholarship from the Iran University of Science and Technology, Tehran, Iran. Asterisk indicates corresponding author. V. Shalchyan is with the Department of Health Science and Technol- ogy, Faculty Medicine, Aalborg University, DK-9220 Aalborg, Denmark, and also with the Department of NeuroRehabilitation Engineering, Bernstein Fo- cus Neurotechnology G¨ ottingen, Bernstein Center for Computational Neu- roscience, University Medical Center G¨ ottingen, Georg-August University, D-37075 G¨ ottingen, Germany (e-mail: vshal@hst.aau.dk). W. Jensen is with the Department of Health Science and Technology, Fac- ulty of Medicine, Aalborg University, DK-9220 Aalborg, Denmark (e-mail: wj@hst.aau.dk). *D. Farina is with the Department of Neurorehabilitation Engineering, Bern- stein Focus Neurotechnology G¨ ottingen, Bernstein Center for Computational Neuroscience, University Medical CenterG¨ ottingen, Georg-August University, D-37075 G¨ ottingen, Germany (e-mail: dario.farina@bccn.uni-goettingen.de). Color versions of one or more of the figures in this paper are available online at http://ieeexplore.ieee.org. Digital Object Identifier 10.1109/TBME.2012.2204991 computer interfacing (BCI). Decoding the discharge pattern of several neurons allows prediction of the motor output. Micro- electrodes can often pick up the action potentials (APs) of a few neurons in a local region near the electrode tip. The sig- nals recorded from these microelectrodes, therefore, contain the spike trains from multiple neural units contaminated by back- ground noise. Retrieving the firing information of different units is the main goal of spike sorting techniques. Such information is not only important for studying brain functions but can also be used as an input for BCI applications. The prerequisite for these studies is detecting the APs in the presence of background noise. The most common method for spike detection is amplitude thresholding which has been often used for real-time imple- mentations of cortically controlled BCI systems [1], [2]. The computational load of this technique is low; however, the pro- cedure is associated with the challenging problem of threshold selection for a tradeoff between false negatives (FNs) and false positives (FPs) [3]. Methods proposed for the automatic identi- fication of the threshold level [4]–[6] are based on the estimation of the background noise power and need prior assumption on the noise amplitude distribution (usually Gaussianity). These assumptions are often not verified [7], [8]. Moreover, an inher- ent problem of the amplitude thresholding methods is that they fail when the spike amplitude peaks are close to or lower than the noise level. Template matching is another approach for extracting the spikes from noisy background. This approach requires the knowledge on the spike shapes [9], [10]. The detection per- formance of this method is higher than simple thresholding; however, as a primary step, in order to form the template of dif- ferent spike morphologies automatically and without any prior knowledge about the signal, another detection algorithm is re- quired which is often based on thresholding [11]–[13], facing similar issues as outlined previously. The nonlinear energy operator (NEO) magnifies local peaks in both amplitude and frequency, and has been widely used for detecting neural spikes [14], [15]. The NEO spike detection method has been reported to perform well and it is attractive because of its easy implementation and computational simplic- ity [16]. A modification on the NEO, called the multiresolution Teager energy operator (MTEO) [17], combines the results of applying the energy operator to the signal with different resolu- tion scales and has shown encouraging results. However, both NEO and MTEO are also threshold-based methods and need manual or automatic level adjustments [18], [19]. 0018-9294/$31.00 © 2012 IEEE