Harnessing GANs for Addition of New Classes in VSR Yaman Kumar 1,2,* , Shubham Maheshwari 2,* , Dhruva Sahrawat 2,* , Praveen Jhanwar 2,* , Vipin Chaudhary 2,* , Rajiv Ratn Shah 2 , and Debanjan Mahata 2,3 1 Adobe 2 MIDAS Lab, IIIT Delhi 3 Bloomberg * Contributed Equally Abstract It is an easy task for humans to learn and gen- eralize a problem, perhaps it is due to their ability to visualize and imagine unseen ob- jects and concepts. The power of imagina- tion comes handy especially when interpolat- ing learnt experience (like seen examples) over new classes of a problem. For a machine learn- ing system, acquiring such powers of imagina- tion are still a hard task. We present a novel ap- proach to low-shot learning that uses the idea of imagination over unseen classes in a classi- fication problem setting. We combine a classi- fier with a ‘visionary’ (i.e., a GAN model) that teaches the classifier to generalize itself over new and unseen classes. This approach can be incorporated into a variety of problem settings where we need a classifier to learn and gen- eralize itself to new and unseen classes. We compare the performance of classifiers with and without the visionary GAN model helping them. 1 Introduction The chief limitation of any deep-learning system is the requirement of huge amount of data. One needs an appreciable amount of data for each class, present in a dataset, in order to make a deep learning system work on all the classes. In fact, deep learning practitioners often try to equalize the distribution of data in various classes in order to not bias a model towards predicting any specific class. However, there are often a few limitations one has to face with datasets. One, for the rare classes (like extinct animals in an animal image dataset) it is hard to get access to some or even any data. Two, it is generally a herculean task to form an exhaustive dataset consisting of all the possi- ble classes for a particular problem. For instance, getting hold of images of all the possible animals in an animal image dataset might not be possible without significant investment of time, money and energy. In 2014, (Goodfellow et al., 2014) presented the idea of GANs. Since then, there has been much research in the domain of GANs which has led to many applications. In their seminal paper, (Goodfellow et al., 2014) mentioned an applica- tion of GANs in the augmentation of a dataset. This has led to some research work motivated to increase the dataset sizes by using GANs (Anto- niou et al., 2017; Springenberg, 2015; Wang et al., 2018). However, most of the works till now chiefly focuses on using GANs as an alias to the tradi- tional approaches of dataset augmentation (like rotation, reflection, cropping, translation, scaling and adding Gaussian noise) (Antoniou et al., 2017; Krizhevsky et al., 2012). This way, one gets a few GAN generated examples which are very similar to the instances present originally in the dataset, thus increasing the dataset size. However, none have so far worked on introducing a completely new class to a dataset. We, in this work, intro- duce a new research direction, where we manufac- ture instances of some entirely new unseen classes in a dataset. We show the efficacy of the gener- ated instances in enabling a deep learning system to predict the unseen classes. For the validation of this approach, we use a downstream applica- tion of Visual Speech Recognition (VSR), com- monly known as lipreading. If the method works, it has the potential of leading to research advances where one needs a small dataset to bootstrap a deep learning system. For the introduction and ad- dition of new classes, the bootstrapped deep learn- ing system may hallucinate (Wang et al., 2018) new instances belonging to the unseen classes and then train itself on those new instances. This way, the model can enable itself for predicting even the unseen classes (which were not originally present in the dataset). arXiv:1901.10139v2 [cs.LG] 30 Jan 2019