A Comparison of Sequence-Trained Deep Neural Networks and Recurrent Neural Networks Optical Modeling for Handwriting Recognition Th´ eodore Bluche 12 , Hermann Ney 23 , and Christopher Kermorvant 1 1 A2iA SA, Paris, France 2 LIMSI CNRS, Spoken Language Processing Group, Orsay, France 3 RWTH Aachen University, Human Language Technology and Pattern Recognition, Aachen, Germany Abstract. Long Short-Term Memory Recurrent Neural Networks are the current state-of-the-art in handwriting recognition. In speech recog- nition, Deep Multi-Layer Perceptrons (DeepMLPs) have become the standard acoustic model for Hidden Markov Models (HMMs). Although handwriting and speech recognition systems tend to include similar com- ponents and techniques, DeepMLPs are not used as optical model in unconstrained large vocabulary handwriting recognition. In this paper, we compare Bidirectional LSTM-RNNs with DeepMLPs for this task. We carried out experiments on two public databases of multi-line hand- written documents: Rimes and IAM. We show that the proposed hybrid systems yield performance comparable to the state-of-the-art, regard- less of the type of features (hand-crafted or pixel values) and the neural network optical model (DeepMLP or RNN). Keywords: Handwriting Recognition •Recurrent Neural Networks •Deep Neural Networks 1 Introduction Handwriting recognition is the problem of transforming an image into the text it contains. Unlike Optical Character Recognition (OCR), segmenting each char- acter is difficult, mainly due to the cursive nature of handwriting. One usually prefers to recognize whole words or lines of text, i.e. the sequence of characters, with HMMs or RNNs. In HMMs, the characters are modeled as sequences of hidden states, associ- ated with an emission probability model. Gaussian Mixture Models (GMMs) is the standard optical model in HMMs. However, in the last decade, emission prob- ability models based on artificial neural networks have (re)gained considerable interest in the community, mainly due to the deep learning trend in computer vision and speech recognition. In this latter domain, major improvements have been observed with the introduction of deep neural networks. A significant usage of neural network for handwriting recognition should also be noted. The MNIST database of handwritten digits received a lot of