SPEECH RECOGNITION WITH DEEP RECURRENT NEURAL NETWORKS Alex Graves, Abdel-rahman Mohamed and Geoffrey Hinton Department of Computer Science, University of Toronto ABSTRACT Recurrent neural networks (RNNs) are a powerful model for sequential data. End-to-end training methods such as Connec- tionist Temporal Classification make it possible to train RNNs for sequence labelling problems where the input-output align- ment is unknown. The combination of these methods with the Long Short-term Memory RNN architecture has proved particularly fruitful, delivering state-of-the-art results in cur- sive handwriting recognition. However RNN performance in speech recognition has so far been disappointing, with better results returned by deep feedforward networks. This paper in- vestigates deep recurrent neural networks, which combine the multiple levels of representation that have proved so effective in deep networks with the flexible use of long range context that empowers RNNs. When trained end-to-end with suit- able regularisation, we find that deep Long Short-term Mem- ory RNNs achieve a test set error of 17.7% on the TIMIT phoneme recognition benchmark, which to our knowledge is the best recorded score. Index Terms— recurrent neural networks, deep neural networks, speech recognition 1. INTRODUCTION Neural networks have a long history in speech recognition, usually in combination with hidden Markov models [1, 2]. They have gained attention in recent years with the dramatic improvements in acoustic modelling yielded by deep feed- forward networks [3, 4]. Given that speech is an inherently dynamic process, it seems natural to consider recurrent neu- ral networks (RNNs) as an alternative model. HMM-RNN systems [5] have also seen a recent revival [6, 7], but do not currently perform as well as deep networks. Instead of combining RNNs with HMMs, it is possible to train RNNs ‘end-to-end’ for speech recognition [8, 9, 10]. This approach exploits the larger state-space and richer dy- namics of RNNs compared to HMMs, and avoids the prob- lem of using potentially incorrect alignments as training tar- gets. The combination of Long Short-term Memory [11], an RNN architecture with an improved memory, with end-to-end training has proved especially effective for cursive handwrit- ing recognition [12, 13]. However it has so far made little impact on speech recognition. RNNs are inherently deep in time, since their hidden state is a function of all previous hidden states. The question that inspired this paper was whether RNNs could also benefit from depth in space; that is from stacking multiple recurrent hid- den layers on top of each other, just as feedforward layers are stacked in conventional deep networks. To answer this ques- tion we introduce deep Long Short-term Memory RNNs and assess their potential for speech recognition. We also present an enhancement to a recently introduced end-to-end learning method that jointly trains two separate RNNs as acoustic and linguistic models [10]. Sections 2 and 3 describe the network architectures and training methods, Section 4 provides exper- imental results and concluding remarks are given in Section 5. 2. RECURRENT NEURAL NETWORKS Given an input sequence x =(x 1 ,...,x T ), a standard recur- rent neural network (RNN) computes the hidden vector se- quence h =(h 1 ,...,h T ) and output vector sequence y = (y 1 ,...,y T ) by iterating the following equations from t =1 to T : h t = H (W xh x t + W hh h t−1 + b h ) (1) y t = W hy h t + b y (2) where the W terms denote weight matrices (e.g. W xh is the input-hidden weight matrix), the b terms denote bias vectors (e.g. b h is hidden bias vector) and H is the hidden layer func- tion. H is usually an elementwise application of a sigmoid function. However we have found that the Long Short-Term Memory (LSTM) architecture [11], which uses purpose-built memory cells to store information, is better at finding and ex- ploiting long range context. Fig. 1 illustrates a single LSTM memory cell. For the version of LSTM used in this paper [14] H is implemented by the following composite function: i t = σ (W xi x t + W hi h t−1 + W ci c t−1 + b i ) (3) f t = σ (W xf x t + W hf h t−1 + W cf c t−1 + b f ) (4) c t = f t c t−1 + i t tanh (W xc x t + W hc h t−1 + b c ) (5) o t = σ (W xo x t + W ho h t−1 + W co c t + b o ) (6) h t = o t tanh(c t ) (7) where σ is the logistic sigmoid function, and i, f , o and c are respectively the input gate, forget gate, output gate and arXiv:1303.5778v1 [cs.NE] 22 Mar 2013