HIGH-RESOLUTION NIR PREDICTION FROM RGB IMAGES: APPLICATION TO PLANT PHENOTYPING Ankit Shukla † , Avinash Upadhyay † , Manoj Sharma † , Viswanathan Chinnusamy ‡ , Sudhir Kumar ‡ † Bennett University, ‡ Indian Agriculture Research Institute ABSTRACT In contrast to the conventional RGB cameras, Near-infrared (NIR) spectroscopy provides images with rich information concerning the biological process of plants. However, NIR spectroscopy is a costly affair and produces low-resolution (LR) images. In this context, recently deep learning-based methods have been proposed in computer vision. In addi- tion, the development of phenomics facilities has facilitated the generation of large plant data necessary for the utilization of these deep learning-based methods. Motivated by these developments, we propose a novel attention-based pix-to-pix generative adversarial network (GAN) followed by a super- resolution (SR) module to generate high-resolution (HR) NIR images from corresponding RGB images. An experiment in- cluding extraction of phenotypic data based on HR NIR im- ages has also been conducted to evaluate its efficacy from an agricultural perspective. Our proposed architecture achieved state-of-the-art performance in terms of MRAE and RMSE on the Wheat plant multi-modality dataset. Index Terms— NIR Prediction, Attention, CNN, Plant Phenotyping, Deep learning, Pix-to-pix 1. INTRODUCTION The demand for food production is skyrocketing due to the growing population. It gives rise to the need for smart plant phenotyping so that we can do better drought analysis, dis- ease detection, and inspection of root growth etc. To avoid crop waste and to get optimum food production rate, the ex- traction of accurate information concerning a plant’s biolog- ical process and crop parameters is important for smart phe- notyping. Traditional methods of phenotyping were manual or semi-automatic. These procedures were destructive and arduous to approach [1]. Hence, image-based plant pheno- typing was introduced and has become an efficient alternative to traditional methods. Information concerning the biological process and crop parameters required for smart plant pheno- typing are not visible by RGB images or human eyes. NIR imaging is a more common way of extracting the above infor- mation. The NIR images for plant phenotyping are captured by multispectral cameras, but these cameras are expensive and have less spatial resolution than RGB cameras. To over- come this issue, [2], [3] proposed machine learning models to predict NIR images from the corresponding RGB images for numerous computer vision applications. These RGB to NIR conversion methods can be utilized for smart plant phenotyp- ing. Some efforts [4], [5] has been done in this direction. Masoomeh Aslahishahri et al. [4] have investigated the GAN-based approach for predicting NIR from visible spec- troscopy for aerial images of the crop. Tian Sun et al. [2] have solved the opposite problem. They have proposed an asymmetric Cycle GAN for NIR to RGB translation to ob- tain colour information. Daniel et al. [6] have proposed a k- nearest neighbour-based approach for predicting agriculture NIR images from RGB areal images. Yumi Iwashita et al.[7] have proposed multiple U-net for NIR estimation from ter- rain RGB imagery collected by satellite data. The utilization of RGB to NIR prediction algorithms for smart plant pheno- typing is extremely limited. There is an immense scope to develop deep learning frameworks for accurate and precise RGB to NIR prediction so we can visualize drought signa- tures and calculate other parameters necessary to understand plant biological processes. This will lead us in the direction of intelligent plant phenotyping. With the advancement in deep learning methods for com- puter vision applications and the development of phenomics facilities have made the generation of large plant data neces- sary for the utilization of these deep learning-based methods quite easier. These facts motivate us to propose a novel attention-based pix-to-pix GAN followed by a super-resolution (SR) module to predict high-resolution (HR) NIR images from its coun- terpart RGB images. Our proposed framework first registers unpaired RGB NIR images. Further, pix-to-pix GAN learns the RGB-NIR transform using these registered pairs of im- ages. NIR images obtained from pix-to-pix GAN are super- resolved using a CNN model in a deep internal learning set- ting as described in [8] to improve the spatial resolution. An experiment including extraction of phenotypic data based on HR NIR images has also been conducted to evaluate its effi- cacy from an agricultural perspective. Our proposed architec- ture achieved state-of-the-art performance in terms of MRAE and RMSE on the wheat plant multi-modality dataset. The main contributions of this paper are as follows: 1) Attention-based pix-to-pix GAN followed by SR module 4058 978-1-6654-9620-9/22/$31.00 ©2022 IEEE ICIP 2022 2022 IEEE International Conference on Image Processing (ICIP) | 978-1-6654-9620-9/22/$31.00 ©2022 IEEE | DOI: 10.1109/ICIP46576.2022.9897670