Using Optimal Embeddings to Learn New Intents with Few Examples: An Application in the Insurance Domain Shailesh Acharya ∗ American Family Insurance, Machine Learning Research Group sachary1@amfam.com Glenn Fung American Family Insurance, Machine Learning Research Group gfung@amfam.com ABSTRACT The ubiquitous adoption of Conversational Agents (CA) in com- mercial settings is changing the way industries interact with their customers. Intent classifcation is an important frst step in de- signing an efcient CA. Every intent that the CA can recognize is represented by a set of natural language examples that are used by the system to learn how to map any user’s utterance to the corre- sponding intent. However, when a new intent is introduced, there are usually not enough examples to train the intent appropriately. In this paper we propose a hybrid system that combines a traditional Deep Neural Network-based classifcation approach with few-shot learning strategies. The simple but yet efective proposed approach achieves good performance for newly introduced intents with few training examples while maintaining performance for previously known intents. We show the potential of the proposed approach on a data generated by a deployed chat system for the insurance do- main. To demonstrate that the propose approach can generalize to other domains, we also perform experiments in a publicly available dataset where we obtain similar approach-substantiating results. CCS CONCEPTS · Computing methodologies → Natural language processing; Learning latent representations. KEYWORDS Dataset, Fewshot learning, Embedding, Chatbot, Intent Classifca- tion Copyright © 2020 for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0). 1 INTRODUCTION Conversational Agent and chatbots are getting increasingly pop- ular in the industry that frequently interact with customers. A successful conversational agent can alleviate the burden on cus- tomer representatives by understanding the customer’s query (of- ten presented in natural language) and guiding the user towards a solution. Intent classifcation is an important frst step in de- signing an intelligent chatbot. It allows chatbot to understand the intent of customer and drive the conversation. For example, if a cus- tomer of an insurance company asks What is the minimum liability Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for proft or commercial advantage and that copies bear this notice and the full citation on the frst page. Copyrights for third-party components of this work must be honored. For all other uses, contact the owner/author(s). KDD Converse’20, August 2020, © 2020 Copyright held by the owner/author(s). limit for state of Wisconsin? then the general intent or superclass that generalizes this query or utterance can be something like AUTO_COVERAGE_LIABILITY_LIMIT. The chatbot should be able to identify the intent and get the right answer from the knowledge base. These intents are defned by business and added/modifed ac- cording to business needs. Every time a new product/service is launched, new intents relating to that service are added to the chatbot. For instance; if a new discount scheme is introduced for usage-based insurance (UBI) then an associated intent for the chat- bot can be UBI_ELIGIBILITY. There could be more than one intent associated with the new service/ofering depending on the scope or complexity of it. Often, the newly added intents could be of signif- cant importance to the business for two reasons; frst, it could be associated to a newly introduced łhot" popular service or ofering and hence much more likely to be queried or solicited by customers and second, there could be a greater business drive to upsell this newly introduced service. For these reasons, it is very important for the chatbot to do well in the intent category/categories relating to this service. An underlying problem with the newly added intents is the lack of enough training examples to train an accurate intent classifer since there is no past interaction with customers regarding this topic. Normally, a subject matter expert comes up with diferent ways of asking questions about that service in order to gather several training samples for that intent class. Such a collection of training examples is limited in number and variation. This can present some challenges to the intent prediction classifer; in spite of having few training examples to begin with, the likelihood of getting queries related to this intent could be high. Furthermore, the cost of misclassifying examples belonging to this new intent class could be potentially much higher to the business. Intent classifcation is usually formulated as a multiclass (some- times multilabel) classifcation problem and the use of deep-neural- network-based classifers is very popular. A well-known trait of training deep neural networks (NN) classifers is that they need large amounts of labeled training data to provide satisfactory classi- fcation performance. They generally perform poorly on categories with fewer training examples. Hence, the ability of deep NN to extract complex statistics and learn high level features from vast datasets is proven. However, most current deep learning approaches have poor sample efciency in contrast to human perception - even a child could recognise a bird after previously seeing a few bird pictures. There are abundant recent works in few-shot learning [7, 8]: an area of machine learning dedicated to solving problems with very few examples per class (but large number of class labels).