Segmentation of cell clusters in Pap smear images using intensity variation between superpixels Marina E. Plissiti 1 , Michalis Vrigkas 1 and Christophoros Nikou 1 1 Department of Computer Science & Engineering, University of Ioannina, Ioannina 45110, Greece marina@cs.uoi.gr Abstract—The automated interpretation of Pap smear images is a challenging issue with several aspects. The accurate segmenta- tion of the structuring elements of each cell is a crucial procedure which entails in the correct identification of pathological situa- tions. However, the extended cell overlapping in Pap smear slides complicates the automated analysis of these cytological images. In this work, we propose an efficient algorithm for the separation of the cytoplasm area of overlapping cells. The proposed method is based on the fact that in isolated cells the pixels of the cytoplasm exhibit similar features and the cytoplasm area is homogeneous. Thus, the observation of intensity changes in extended subareas of the cytoplasm indicates the existence of overlapping cells. In the first step of the proposed method, the image is tesselated into perceptually meaningful individual regions using a superpixel algorithm. In a second step, these areas are merged into regions exhibiting the same characteristics, resulting in the identification of each cytoplasm area and the corresponding nuclei. The area of overlap is then detected using an algorithm that specifies faint changes in the intensity of the cytoplasm of each cell. The method has been evaluated on cytological images of conventional Pap smears, and the results are very promising. Keywords—Pap smear images, superpixels, cell cluster separa- tion, overlapped cell segmentation. I. I NTRODUCTION Cervical cancer is considered to be the fourth most com- mon cause of cancer death in women worldwide [1]. The im- pact of the well known Pap-test examination in the prevention of cervical cancer is undoubtedly significant. Through this examination precancerous conditions and abnormal changes that may develop into cancer are recognized and they are early treated preventing the development of cervical cancer. The Pap smear slides are examined under a microscope to identify abnormalities in the structure and morphology of cells. In practice, the interpretation and the characterization of the findings in these microscopical images are usually obtained by expert cytologists. The correct classification of a slide and the accurate diagnostic conclusion are crucial for the effective treatment of each incident. However, the limitations exhibited by these images in combination with human error may lead to misclassifications. Thus, many efforts have been made by several researchers for the development of automated methods for the analysis of Pap smear images, in order to assist the diagnostic procedure. This work is co-financed by the European Union (European Regional Development Fund- ERDF) and Greek national funds through the Operational Program THESSALY- MAINLAND GREECE AND EPIRUS-2007-2013 of the National Strategic Reference Framework (NSRF 2007-2013). The structural elements of each cell provide significant information about the pathological condition of the smear. The nuclei features have shown high discriminative ability in recognizing pathological cases [2]. For this reason, many methods are focused on the accurate segmentation of the nuclei. Some of the general approaches that were used for this purpose are morphological analysis [3], watershed transform [4] and contour detectors [5]. Nevertheless, the cytoplasm area may adduce informative facts about the cell, such as the cytoplasmic transparency and the thickness of the cytoplasmic membrane. Recent works are focussed on the segmentation of both the nuclei and the cytoplasm. The first attempts were performed on free-lying cells, where no cell overlapping is occurred [5], [6], [7]. However, as the cell overlapping is a widespread phenomenon in Pap smear images, more sophisticated methods that are able to achieve reliable segmentation of the cytoplasm of each cell in cell clusters were proposed. More specifically, in [8] a geometric active contour based method for the localization of cells and the detection of the nucleus and cytoplasm boundaries is developed. In the method proposed in [9], the cell segmentation is performed using a joint level set optimization on all detected nuclei and cytoplasm pairs. In addition, the locally constrained watershed transform is performed for the separation of overlapped cells in [10]. In this work, we present a novel method for the definition of the area of each individual cell in clusters containing two overlapping cells. The method exploits the similarity in inten- sity of small area fractions belonging to the same structuring element of the cell, in order to concatenate them in a single aggregated item. The area fractions are obtained using a super- pixel algorithm [11]. The final boundaries of each cytoplasm are obtained after the estimation of the boundaries segments in overlapping areas. This is achieved with the detection of the cytoplasm subareas presenting intensity disparity compared to the cytoplasm area without overlap. The experimental results indicate that the proposed method is robust and it provides accurate cell delineation. II. METHOD In order to take advantage of the intensity characteristics of each region of interest, it is convenient to separate the image into perceptually meaningful individual regions. This is achieved with the application of SLIC segmentation algorithm [11]. Thus, each image is tesselated into approximately equally sized subregions, presenting homogeneous intensity character- istics (Fig. 1). Next, we classify the superpixels of the cell 184