A Novel Optimizer in Deep Neural Network for Diabetic Retinopathy Classification Pranamita Nanda 1,* and N. Duraipandian 2 1 Department of Computer Science and Engineering, Velammal Institute of Technology, Chennai, 601204,Tamilnadu, India 2 Department of Computer Science and Engineering, Saveetha Engineering College, Chennai, 602105, Tamilnadu, India *Corresponding Author: Pranamita Nanda. Email: pranamitananda.cse@velammalitech.edu.in Received: 27 October 2021; Accepted: 10 December 2021 Abstract: In severe cases, diabetic retinopathy can lead to blindness. For decades, automatic classification of diabetic retinopathy images has been a challenge. Med- ical image processing has benefited from advances in deep learning systems. To enhance the accuracy of image classification driven by Convolutional Neural Net- work (CNN), balanced dataset is generated by data augmentation method fol- lowed by an optimized algorithm. Deep neural networks (DNN) are frequently optimized using gradient (GD) based techniques. Vanishing gradient is the main drawback of GD algorithms. In this paper, we suggest an innovative algorithm, to solve the above problem, Hypergradient Descent learning rate based Quasi hyper- bolic (HDQH) gradient descent to optimize the weights and biases. The algo- rithms only use first order gradients, which reduces computation time and storage space requirements. The algorithms do not require more tuning of the learning rates as the learning rate tunes itself by means of gradients. We present empirical evaluation of our algorithm on two public retinal image datasets such as Messidor and DDR by using Resnet18 and Inception V3 architectures. The find- ings of the experiment show that the efficiency and accuracy of our algorithm out- performs the other cutting-edge algorithms. HDQHAdam shows the highest accuracy of 97.5 on Resnet18 and 95.7 on Inception V3 models respectively. Keywords: CNN; diabetic retinopathy; data augmentation; gradient descent; deep learning; optimization 1 Introduction Diabetic retinopathy (DR) is the most common complication of diabetes. It is the major cause of permanent blindness in people in their working years [1]. Inflammation and retinal neurodegeneration, in addition to microvascular alterations, may contribute to diabetic retinal damage in the early stages of DR. DR affects about 100 million people worldwide and is expected to become a growing burden, with estimates showing that DR-related visual impairment and blindness increased by 64 percent and 27 percent, respectively, between 1990 and 2010 [2]. Mild Non-Proliferative Diabetic Retinopathy (NPDR), moderate NPDR, severe NPDR, and PDR are the four phases of DR in terms of severity. This work is licensed under a Creative Commons Attribution 4.0 International License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited. Computer Systems Science & Engineering DOI: 10.32604/csse.2022.024695 Article ech T Press Science