Size-Invariant Cell Nucleus Segmentation in 3-D Microscopy Sundaresh Ram ∗ , Jeffrey J. Rodr´ ﾆguez ∗ and Giovanni Bosco † . ∗ Department of Electrical and Computer Engineering, University of Arizona, Tucson, AZ, USA. † Department of Molecular and Cellular Biology, University of Arizona, Tucson, AZ, USA. Abstract—Accurate segmentation of 3-D cell nuclei in mi- croscopy images is an essential task in many biological studies. Traditional image segmentation methods are challenged by the complexity and variability of microscope images, so there is a need to improve segmentation accuracy and reliability, as well as the level of automation. In this paper we present a novel auto- mated algorithm for robust segmentation of 3-D cell nuclei using a combination of ideas. Our algorithm includes the following steps: image denoising, binarization, seed detection using the fast radial symmetric transform (FRST), initial segmentation using the random walker algorithm and the 3-D watershed algorithm, and ſnal reſnements using 3-D active contours. We show that our algorithm provides improved accuracy compared to existing segmentation algorithms. Index Terms—Active contours, random walker algorithm, watershed algorithm. I. I NTRODUCTION The 3-D segmentation and reconstruction of cells in ƀuores- cent in-situ hybridization (FISH) images is an important ſrst task in counting cells as well as for many biological studies including high throughput analysis of gene expression level, morphology, quantifying molecular markers and phenotypes in a single cell level. Also cell migrations and deformations play an essential role in biological processes, such as parasite inva- sion, immune response, embryonic development, and cancer [2]. Thus, there is a signiſcant interest in these applications to be able to detect and segment the 3-D cell nuclei with high accuracy. Manual segmentation of these cells is difſcult, time con- suming and error-prone. This is due to various reasons such as ƀuctuating intensities and morphological variations of the cells within the images, cell nucleus spanning across multiple 2-D images, closely clustered cells which are touching in 3-D, cells of varying shapes and sizes, accidental and non-speciſc staining, highly textured nuclei due to variable cromatin struc- ture, low signal-to-noise ratio (SNR) in these images, spectral unmixing errors, microscopy imaging limitations and the large number of images that the pathologist has to perform cell segmentation on. Moreover, manual analysis is user-dependent in a clinical setting, as the task can be subjective and speciſc training is often lacking. These factors serve as a motivation for designing an automated segmentation algorithm for 3-D cell segmentation. However, due to the reasons mentioned above, design of an automated/semi-automated segmentation algorithm for this task is challenging. Cell segmentation is a widely studied topic with numer- ous algorithms available in the literature. Dufour et al. [2] proposed a cell segmentation and tracking scheme based on multiple active surfaces, coupled by a penalty for overlaps. This method does not perform well when the cells are highly textured and a wrong initialization of the active surface can lead to errors in segmentation. Li et al. [8] proposed an automated algorithm for cell segmentation involving a tracking procedure on the diffused gradient vector ƀow ſeld obtained by an elastic deformation and local adaptive thresholding. This technique requires that the centers of the nuclei be brighter than nearby surrounding regions and will suffer in case of textured nuclei. Dzyubachyk et al. [3] proposed several im- provements to Dufour’s algorithm such as the use of the Radon transform to “decouple” the active surfaces of touching cells. Using this method the clustered cells are always partitioned via a plane, which may not be the actual boundary separating them. Recently Al-Kofahi et al. [6] proposed an algorithm for cell segmentation based on graph-cuts segmentation. This method requires that the input images have a bimodal his- togram, and for large cells it will ſnd multiple seeds within one nucleus. In this paper we propose a new automated technique to segment the 3-D cell nuclei of Drosphila melanogaster ob- tained via confocal microscopy. We use the multiscale variance stabilizing transform (MS-VST) proposed by Zhang et al. [15] to remove the effect of noise, binarize the images using histogram thresholding, ſnd seeds or markers (one for each nucleus) using the fast radial symmetric transform (FRST) and non-maximum suppression, obtain an initial segmentation using the random walker segmentation algorithm with the detected seeds and 3-D watershed algorithm and use 3-D active contours as a ſnal reſnement. II. METHODS The ovarian germline of the Drosophila melanogaster con- sists of two types of cells, namely “nurse cells” and “follicle cells”. The follicle cells are smaller than the nurse cells and surround the nurse cells in an ellipsoidal fashion in 3-D. Fig. 1a shows a single slice of a 3-D data set where the nurse cells are surrounded by the follicle cells in an elliptical shape. A. Denoising and Thresholding We use Zhang’s MS-VST [15] as a preprocessing step to remove the background noise in each slice, thereby increasing the contrast between the foreground nuclei and the surrounding background. An example image after denoising is shown in Fig. 1b. The MS-VST denoising enhances the separation between modes in the bimodal histogram of each image slice. For our image data, the foreground nuclei regions are present 37 978-1-4673-1830-3/12/$31.00 ©2012 IEEE SSIAI 2012