International Journal of Computing and Digital Systems ISSN (2210-142X) Int. J. Com. Dig. Sys.13, No.1 (Apr-23) http://dx.doi.org/10.12785/ijcds/130170 Deep Learning Model for Automatic Detection of Oral squamous cell carcinoma (OSCC) using Histopathological Images Sayyada Hajera Begum 1 and P Vidyullatha 2 1 Research Scholar,Department of Computer Science and Engineering,Koneru Lakshmaiah Education Foundation, Vaddeswaram, AP, India 2 Associate Professor,Department of Computer Science and Engineering,Koneru Lakshmaiah Education Foundation, Vaddeswaram, AP, India Received 2 May. 2022, Revised 21 Dec. 2022, Accepted 6 Feb. 2023, Published 16 Apr. 2023 Abstract: Oral squamous cell carcinoma (OSCC), is a type of cancer that causes the loss of the structural formation of layers and membranes in the oral cavity region. With the recent advent of Deep learning (DL) in biomedical image classification, the automated early diagnosis of oral histopathological images can aid in effective treatment of oral cancer. This work attempts to perform an automated classification of benign and malignant oral biopsy histopathological images by implementing a DL-based convolutional neural network (CNN) model for the initial analysis of OSCC. For this research, four recently developed candidate pre-trained DL-CNN models namely NASNetLarge, InceptionNet, Xception, and DenseNet201 are selected through the approach of transfer learning. These pre-trained models are then modified with additional layers for effective OSCC detection. The efficacy of these modified models is examined on an oral cancer histopathological image database. It is examined that the pre-trained DenseNet201 model with modified structure has surpassed other models in terms of performance parameters by recording an accuracy of 91.25% and is considered as our proposed DL-CNN model. Keywords: oral cancer detection, Oral squamous cell carcinoma (OSCC), Deep Learning (DL), Convolutional Neural Network (CNN). 1. INTRODUCTION The rate of oral cancer is known to be highest worldwide and the incidence is lower in women compared to men and nearly 660,000 new incidences of oral cancer are reported each year and more than 340,000 deaths worldwide due to lack of timely diagnosis. In oral cancer, the cancerous tissues can be located in the lips, oral cavity, and pharynx and causes the loss of the structural formation of layers and membranes in the oral cavity region. Oral cancers are classified into OSCC, salivary gland carcinoma, verrucous carcinoma, and lymphoepithelial car- cinoma. The majority of the carcinomas are due to OSCC [1], [2]. Despite applying various treatment modalities, the total mortality rate of OSCC is not declined significantly which is only due to lack of efforts for early detection and diagnosis. The physicians examine the presence of any suspicious le- sion which can be cancerous and suggests for biopsy. Slides with the biopsy sections are observed for any deformities which are different from usual cell arrangements like size and shape using microscope [3]. At the histopathological level, malignant squamous cells are bigger compared to the normal cells and are particularly different from each other in shapes. A confirmatory diagnosis of oral cancer from this report is needed to be done by a highly qualified and experienced specialist which is very vital and needs to be accurate [3]. However, the entire manual data interpretation of the cancerous slide is too time-consuming and at the same time is prone to human errors [4]. Because of the above-mentioned reasons, computer- aided diagnostic (CAD) techniques may assist the physi- cians in reducing both time and bias with improved effi- ciency in the analysis of the features. The intention is to discover cancer at an early stage which will lead to early treatment, which lowers the risk of morbidity and mortality. Moreover, the oral diagnosis CAD systems will reduce the volume of load in the laboratories and most of the cases may be benign, the pathologist may focus more on malignant cases [5] . In the development of CAD systems, biomedical imag- E-mail address: sayyada.hajera07@gmail.com http:// journals.uob.edu.bh