TEM Journal. Volume 12, Issue 2, pages 813-819, ISSN 2217-8309, DOI: 10.18421/TEM122-26, May 2023. TEM Journal – Volume 12 / Number 2 / 2023. 813 Model of Watershed Segmentation in Deep Learning Method to Improve Identification of Cervical Cancer at Overlay Cells Dwiza Riana 1,2 , Muh Jamil 1 , Sri Hadianti 1 , Jufriadif Na’am 1 , Hadi Sutanto 2,3 , Ronald Sukwadi 2,4 1 Universitas Nusa Mandiri, Jalan Raya Jatiwaringin No 2, East Jakarta, Indonesia 2 Universitas Katolik Indonesia Atma Jaya, Program Profesi Insinyur, Jalan Jendral Sudirman No.51, South Jakarta, Indonesia 3 Atma Jaya Catholic University of Indonesia, Mechanical Engineering Masters Program, Jalan Jendral Sudirman No.51, South Jakarta, Indonesia 4 Atma Jaya Catholic University of Indonesia, Industrial Engineering Study Program, Jalan Jendral Sudirman No.51, South Jakarta, Indonesia Abstract – Cervical cancer is a disease that is very scary for women because it is the cause of death among women. To be aware of this disease is to do an early examination through the Pap Smear (PS) test. In terms identifying overlapping cancer cells, it still has low accuracy. Therefore, this research was carried out with the aim of getting the level of cell separation with high accuracy. This study uses a model to develop the Watershed segmentation technique in the Deep Learning Method. The data tested in this study comes from the RepomedUNM dataset. The amount of data tested is 420 overlapping images with the formulation of 1,260 test images. The results of this study can very well separate each overlapping cell with an average Intersection over Union (IoU) score of 0.9061. Each result can be divided fully by the whole of its area, so the final results of overlapping cells were successfully separated with an average score of 0.945. Therefore, this research can be used as a reference in identifying cervical cancer cells. DOI: 10.18421/TEM122-26 https://doi.org/10.18421/TEM122-26 Corresponding author: Dwiza Riana, Universitas Nusa Mandiri, Jalan Raya Jatiwaringin No 2, East Jakarta, Indonesia Email: dwiza@nusamandiri.ac.id Received: 28 November 2022. Revised: 16 March 2023. Accepted: 10 April 2023. Published: 29 May 2023. © 2023 Dwiza Riana, Muh Jamil, Sri Hadianti, Jufriadif Na’am, Hadi Sutanto, Ronald Sukwadi; published by UIKTEN. This work is licensed under the Creative Commons Attribution-NonCommercial-NoDerivs 4.0 License. The article is published with Open Access at https://www.temjournal.com/ Keywords – cervical cancer, Pap Smear, segmentation, deep learning, overlay cell. 1. Introduction Cervical cancer (CC) is a disease that requires apprehension with it [1]. This cancer is one of the leading causes of death in women globally [2]. Most of the cells are relatively thin and lie beneath the surrounding tissue [3], making them very difficult to be identifed [4]. The death rate due to CC can be significantly reduced by carrying out an early examination through the Pap Smear (PS) test [5]. The images from the results of these tests can be observed for abnormal cell conditions [6], but they tend to be very troublesome and are prone to errors when inspected manually [7]. For this reason, digital image processing techniques are needed to assist inspections so that high accuracy results are obtained. There are still weaknesses in existing techniques, resulting in low accuracy for some cell classes [8]. The use of segmentation techniques for overlapping cells is still low [9]. Therefore, research in developing segmentation techniques to identify overlapping cells in CC images is needed. Cell segmentation on PS images is very important in identifying pre-cancerous changes in CC [10]. Several studies to segment CC cells have been carried out. The proposed method can work automatically or semi-automatically. Segmentation using the Mean-Shift clustering algorithm and Mathematical Morphology can identify cervical cell nuclei very effectively [11]. Selective-Edge- Enhancement-based Nuclei Segmentation method (SEENS) can achieve higher accuracy in cervical core segmentation [12].