Deep Neural Networks for Text: A Review Chiung Ching Ho 1 , Khairul Nizam Baharim 2 , Ahmad Abdulsalam Ahmad Fatan 2 , and Mohd Shafiq Bin Alias 2 1 Multimedia University, Data Science Institute, 2 TM Research & Development Corresponding author’s email : ccho@mmu.edu.my {khairulnizam,abdulsalam,shafiq}@tmrnd.com.my Abstract: Deep learning is a powerful technique for learning representation and can be used to learn features within text. The learned features are useful for solving Natural Language Processing problem. In this paper we review key literature related to deep learning and its application on solving text analysis. Keywords: Deep Learning, Text Analysis, CNN, RNN 1. Introduction Text data has been an important source of information, and continues to provide information in ever increasing volumes. Deep learning has been applied successfully for the purpose of learning features of text, and can be applied to enable many Natural Language Processing problem. In this paper, we review the application of deep learning onto text, focusing on the convolutional neural network (CNN) architecture. This paper is divided into the following sections: 1.0 Introduction, 2.0 What is Deep Learning , 3.0 Deep Learning and Text, 4.0 Convolution Neural Network and Text, 5.0 Recurrent Neural Network and Text, and 6.0 Conclusion. 2. What is Deep Learning Deep learning [1] first came to public attention when it was used to win the 2012 ImageNet Large-Scale Visual Recognition Challenge with a top-5 test error rate of 15.4% [2]. Recently, AlphaGo used DL to defeat a human professional Go player which was an unprecedented achievement [3]. Deep learning is being increasingly considered as a subset of machine learning, both supervised and unsupervised. In machine learning, human-engineered representations and input features are important to ensure that the machine learning task is performed with high accuracy. The various machine learning algorithms optimizes the weights for the representation and input features to make the best final prediction. In practice, machine learning can thus be divided into two aspect. The first aspect calls for domain-specific subject-matter expertise to describe the features in a way that is suitable for computing. The second aspect is to optimize the weights of these features via a learning algorithm. The deep learning approach builds on the aforementioned ideas by attempting to automatically learn good features or representations, in a process described as representation learning. In the course of representation learning, multiple- levels of representation is learned from ‘raw’ inputs (for example, text) to construct an output. The accuracy of the learned representation is driven by automatically adjusted weights. The application of deep learning as a tool for solving machine learning problems have several advantages over techniques which are highly dependent on human engineered features. Human engineered features are very often highly specific, and are incomplete due to latent patterns which are not detected even by subject matter experts. The process of engineering features itself takes a tremendous amount of time to design and validate. Learned representations and features takes relatively lesser amount of time to be learned, and are easily adaptable. In terms of generalization, the deep learnable framework is flexible and is useful for learning real-world, time-dependent, visual and linguistic information. Deep learning facilitates both unsupervised learning (from raw text for example) and supervised learning (text which are labelled). Applications of deep learning in problem solving has been increasing of late due to the following factors: the growth of available data, advances in multicore CPUs and GPUs which facilitates deep learning, and the increase of new models and algorithms. Figure 1 depicts a generic deep learning network which has n-layers. The layers comprise one input layer, m-hidden layers and one output layer. Figure 1: Generic deep learning network [4] In this paper, we focus our discussion for deep learning on deep neural networks, which is the dominant model within deep learning. We also choose to focus the discussion on natural language processing, instead of a historical accounting of deep neural networks [5].