Contents lists available at ScienceDirect Computers and Electronics in Agriculture journal homepage: www.elsevier.com/locate/compag Late fusion of multimodal deep neural networks for weeds classification Vo Hoang Trong ⁎ ,1 , Yu Gwang-hyun, Dang Thanh Vu, Kim Jin-young ⁎ ,1 Department of Electronics and Computer Engineering, Chonnam National University, Gwangju, Republic of Korea ARTICLE INFO Keywords: Convolutional neural network Bayesian conditional probability Priority weights Voting method Weeds classification ABSTRACT In agriculture, many types of weeds have a harmful impact on agricultural productivity. Recognizing weeds and understanding the threat they pose to farmlands is a significant challenge because many weeds are quite similar in their external structure, making it difficult to classify them. A weeds classification approach with high ac- curacy and quick processing should be incorporated into automatic devices in smart agricultural systems to solve this problem. In this study, we develop a novel classification approach via a voting method by using the late fusion of multimodal Deep Neural Networks (DNNs). The score vector used for voting is calculated by either using Bayesian conditional probability-based method or by determining priority weights so that better DNNs models have a higher contribution to scoring. We experimentally studied the Plant Seedlings and Chonnam National University (CNU) Weeds datasets with 5 DNN models: NASNet, Resnet, Inception–Resnet, Mobilenet, and VGG. The results show that our methods achieved an accuracy of 97.31% on the Plant Seedlings dataset, and 98.77% accuracy on the CNU Weeds dataset. Furthermore, our framework can classify an image in near real- time. 1. Introduction Weeds are one of the most critical factors in the reduction of agri- cultural productivity and increase the burden on farmers’ crops. Correct identification of weeds is an essential step for analyzing the threats of such weeds to secure agricultural productivity. It may be challenging to identify weeds belonging to the same family visually because of their natural similarity. Various applications that rely on physical features to classify weeds, such as the Naive Bayes and Gaussian mixture models, were adopted in De Rainville et al. (2014). Conversely, Persson and Åstrand (2008) applied Active shape models to extract weeds from images and k-nearest neighbors (KNNs) for the classification. However, the handcrafted features were overly simplified and could not empha- size the characteristics of the weeds in real-world applications. Recently, with the improvement of camera quality and the devel- opment of hardware specifications, especially of graphics processing units (GPUs), researchers have achieved significant results when ad- dressing the weeds classification problem by using deep learning models. Dyrmann et al. (2016) built a convolutional neural network (CNN) model based on the concept of the Resnet model. Chavan and Nandedkar (2018) combined the underlying architecture of AlexNet and VGG to form AgroAVNET, which included 6 convolutional and 3 fully connected layers. They applied it to a small dataset (Plant Seedlings Dataset), which includes 12 species and approximately 4200 images from the Aarhus University Computer Vision and Biosystems Signal Processing Group (Giselsson et al., 2017). Their model exhibited impressive performance; however, those deep learning-based methods were not sufficient for the classification of the Plant Seedlings dataset, which includes complex weeds structures. To solve this problem, we propose a novel classification using the voting method with the late fusion of multimodal DNNs. The score method used for voting is calculated by alternately using Bayesian conditional probability-based proposed in Kittler et al. (1998) or by using performances of DNNs models to determine priority weights. The species are then scored considering the weighted linear combination or weighted power multiplication. The better the performance, the higher the priority weight. We tested our voting method experimentally by using the late fusion of 5 DNN models on the CNU and Plant Seedlings weeds datasets. We demonstrate that combining multiple DNNs models and giving priority to the best performing models can provide more accurate classification results in near real-time with central processing unit (CPU) or GPU. The remainder of this paper is organized as follows. Section 2 summarizes previous approaches of weeds classification using fusion and DNNs models. The CNU Weeds and Plant Seedlings datasets are introduced in Section 3. Section 4 describes the overall voting method https://doi.org/10.1016/j.compag.2020.105506 Received 7 October 2019; Received in revised form 28 April 2020; Accepted 12 May 2020 ⁎ Corresponding author. E-mail address: beyondi@chonnam.ac.kr (K. Jin-young). 1 The authors contributed equally to this work. Computers and Electronics in Agriculture 175 (2020) 105506 0168-1699/ © 2020 Elsevier B.V. All rights reserved. T