Scripta Materialia 193 (2020) 33–37 Contents lists available at ScienceDirect Scripta Materialia journal homepage: www.elsevier.com/locate/scriptamat Race against the Machine: can deep learning recognize microstructures as well as the trained human eye? Michiel Larmuseau a,b,c,∗ , Michael Sluydts a,b , Koenraad Theuwissen d , Lode Duprez d , Tom Dhaene c , Stefaan Cottenier a,b a Center for Molecular Modeling, Ghent University, Technologiepark 46, Zwijnaarde, B-9052, Belgium b Department of Electromechanical, Ghent University, Technologiepark 46, Zwijnaarde, B-9052, Belgium c IDLab, Department of Information Technology, Ghent University – IMEC, Technologiepark 126, Zwijnaarde, B-9052, Belgium d OCAS NV/ArcelorMittal Global R&D Gent, Pres. J. F. Kennedylaan 3, Zelzate, B-9060, Belgium a r t i c l e i n f o Article history: Received 23 July 2020 Revised 8 October 2020 Accepted 12 October 2020 Keywords: Image analysis Steels Modeling Scanning electron microscopy (SEM) a b s t r a c t The promising results of deep learning in image recognition suggest a huge potential for microscopic analyses in materials science. One major challenge for its adoption in the study of materials is the lim- ited number of images that are available to train models on. Herein, we present a methodology to create accurate image recognition models with small datasets. By explicitly taking into account the magnifi- cation and by introducing appropriate transformations, we incorporate as many insights from material science in the model as possible. This allows for a highly data-efficient training of complex deep learning models. Our results indicate that a model trained with the presented methodology is able to outperform human experts. © 2020 Acta Materialia Inc. Published by Elsevier Ltd. All rights reserved. Microscopy images are an important source of information on the small-scale structure of materials, referred to as the mi- crostructure. However, most of the time, this information is anal- ysed only qualitatively: domain experts for instance mainly exam- ine these images to assess whether the processing of the material went well. While such an assessment is important, a lot of infor- mation contained in the microscopy image is not used. In recent years, approaches towards a more quantitative analysis of these images using machine learning have been thoroughly investigated in literature[1–4]. However, all these approaches rely on generic microstructure descriptors that are not tailored to the specific ma- terials in the dataset, resulting in sub-optimal performance. Deep learning methods[5] make it possible to learn microstruc- tural descriptors directly from the available data. Despite the first report of deep learning outperforming humans in an image recog- nition task already dating back to 2011[6], its adoption in practical material science remains limited[7–9]. A possible explanation for this, is that deep learning is often only deemed to outperform clas- sical methods when large datasets of images are available. With most commonly used datasets in material science literature con- taining around the order of a thousand images[2,4,10], this opinion feels reasonable. However, recent work[11] has shown that by cre- ∗ Corresponding author. E-mail address: michiel.larmuseau@ugent.be (M. Larmuseau). ating tailor-made deep learning models for specific datasets, one can outperform models that use generic microstructure descrip- tors. This would make it possible to investigate how well deep learn- ing models perform in recognizing microstructures compared to expert materials scientists. Although it is clear from other fields in computer vision that deep learning models can outperform human experts[12,13] provided a sufficient number of images, we here aim to investigate the performance of models that are trained on only around a hundred microstructure images in total. A hundred im- ages is a practical amount, as it can easily be collected in a sys- tematic study of a class of materials. To evaluate the performance of both a neural network that is obtained using the methodology presented in [11] and the panel of experts, we have organized two different quizzes. For the first quiz, 36 microscopy images need to be assigned to one of the five pre-defined classes of microstructures. The option “None of these” is included in case a microstructure image is shown that does not belong to one of the five pre-defined classes. The dataset on which the model was trained, the training set, contains 134 images in to- tal, with magnifications, expressed as pixels per micrometer, rang- ing from 1.1 to 212 pixels per micrometer (ppμm). Both optical mi- croscopy images and scanning electron microscopy (SEM) images have been included. An example for each type of microstructure is shown in Fig. 1 (a). For the second quiz, 21 SEM microscopy images of complex martensitic steels need to be assigned to one https://doi.org/10.1016/j.scriptamat.2020.10.026 1359-6462/© 2020 Acta Materialia Inc. Published by Elsevier Ltd. All rights reserved.