ARTICLE IN PRESS JID: NEUCOM [m5G;June 29, 2019;10:59] Neurocomputing xxx (xxxx) xxx Contents lists available at ScienceDirect Neurocomputing journal homepage: www.elsevier.com/locate/neucom Defective samples simulation through adversarial training for automatic surface inspection Lizhe Liu, Danhua Cao ∗ , Yubin Wu, Taoran Wei School of Optical and Electronic Information, Huazhong University of Science and Technology, 430074 Wuhan, China a r t i c l e i n f o Article history: Received 19 July 2018 Revised 16 April 2019 Accepted 29 May 2019 Available online xxx Communicated by Dr. Li Sheng Keywords: Image simulation Data augmentation Adversarial training Automatic surface inspection Generative adversarial nets a b s t r a c t Deep learning has shown great potential in machine vision. However, introducing deep learning to auto- mated surface inspection is still a challenging task because it depends heavily on the quality of training samples, especially the defective samples. In this paper, a general defective samples simulation method based on generative adversarial nets (GAN) is proposed to deal with the limitation of defective samples in production. Under the GAN framework, the simulative network with encoder–decoder architecture is proposed. Further, the simulative network and discriminative network are trained adversarially under the proposed regional training strategy, which gives priority to the translation of the defective area. Finally, the defect-free area is refined through wavelet fusion. This method requires a small number of defective training samples, and can generate simulative defects of specified shapes and types and meanwhile ob- tain the pixel-wise ground truth. The simulative samples can be used directly for training of deep learning based automated surface inspection tasks. We conduct experiments on four datasets. The experimental results show that our method can generate defective samples of higher quality than general image trans- lation methods. Applying this method to surface defect inspection can significantly improve the effect of defect inspection model based on deep learning. © 2019 Elsevier B.V. All rights reserved. 1. Introduction With the development of machine vision and industrial au- tomation, automated surface inspection (ASI) based on machine vi- sion has been increasingly applied in product quality control and maintenance. Traditional machine vision based inspection methods involve filtering methods, structural methods, statistical methods and model-based methods, which have been successfully applied to the inspection of industrial products and materials such as steel [1], optoelectronic devices [2], textile [3] and roads [4]. With ac- curate feature extraction and operator design, traditional methods can achieve high accuracy and speed in specific inspection tasks, but lack the versatility. In recent years, deep learning technology, which is a new so- lution to this problem, has continuously made breakthroughs [5]. Deep-learning based methods usually adopt end-to-end training and not need design of feature extraction operators and classifiers. Some popular research directions such as image classification and semantic segmentation have strong similarities with automated surface inspection tasks. However, introducing deep learning into surface inspection is challenging, because practical applications ∗ Corresponding author. E-mail address: dhcao@hust.edu.cn (D. Cao). often suffer from insufficient training data. Many researches [6– 8] show that training of deep learning models requires a large number of training samples, otherwise overfitting may easily occur. For example, in natural image classification tasks [9], a training set of a million level is used; in semantic segmentation tasks [10,11], thousands of pixel-labeled training samples are often required. However, in industrial production, defective samples are often scarce and the labeling work is time-consuming. Therefore, making a training set is a very costly job. Generally, there are two approaches to this problem. The first is to limit the complexity of the model to prevent overfitting, thereby reducing the need for training data. This is a common way with many related studies. For instance, Ren et al. [12] proposed to use networks pre-trained on large datasets such as ImageNet [9], fix the convolutional parameters, and only fine-tune the last few lay- ers. Natarajan et al. [13] directly utilize the pre-trained CNN model without any fine-tuning to extract features which are used as the input of traditional classifiers such as SVM. Experiments indicate that this way can achieve good results on small training datasets. Another approach is to augment training data to increase the sample diversity. Traditional data augmentation approaches includ- ing cropping, rotation, mirroring, and color-shift, have been used as general steps for data preprocessing of deep learning and widely adopted in many defect inspection methods [14–16]. However, the above methods can only be amplified on the basis of the original https://doi.org/10.1016/j.neucom.2019.05.080 0925-2312/© 2019 Elsevier B.V. All rights reserved. Please cite this article as: L. Liu, D. Cao and Y. Wu et al., Defective samples simulation through adversarial training for automatic surface inspection, Neurocomputing, https://doi.org/10.1016/j.neucom.2019.05.080