Biomedical Signal Processing and Control 8 (2013) 30–40 Contents lists available at SciVerse ScienceDirect Biomedical Signal Processing and Control j o ur nal homep a ge: www.elsevier.com/locate/bspc FISH image analysis using a modified radial basis function network Christos Sagonas a , Ioannis Marras a,∗ , Ioannis Kasampalidis a , Ioannis Pitas a , Kleoniki Lyroudia b , Georgia Karayannopoulou c a Aristotle University of Thessaloniki, Department of Informatics, Box 451, 54124 Thessaloniki, Greece b Aristotle University of Thessaloniki, Department of Endodontology, Dental School, 54124 Thessaloniki, Greece c Aristotle University of Thessaloniki, Department of Pathology, Medical School, 54124 Thessaloniki, Greece a r t i c l e i n f o Article history: Received 22 October 2011 Received in revised form 26 April 2012 Accepted 14 May 2012 Available online 2 June 2012 Keywords: FISH Cell nucleus segmentation Spot detection HER-2/neu Gene amplification a b s t r a c t Fluorescent in situ hybridization (FISH) is an exceptionally useful method in determining HER-2/neu gene status in breast carcinoma samples, which is a valuable cancer prognostic indicator. Its visual evaluation is a difficult task, which involves manual counting of red/green dots in multiple microscopy images, a procedure which is both time consuming and prone to human errors. A number of algorithms have recently been developed dealing with the (semi)-automated analysis of FISH images. Their performance is quite promising, but further improvement is required in their diagnostic accuracy. In addition, they have to be evaluated on large FISH image data sets. Here, we present a novel method for analyzing FISH images based on cell nuclei and red/green spot modelling by radial basis functions (RBFs). Our method was compared to one of the most prominent methods reported in the literature on a large data set, comprised of 246 breast cancer cases (in total 3412 FISH images) and showed statistically significant diagnostic accuracy improvement, especially on HER-2/neu positive cases. The overall diagnostic accuracy of the proposed method is 95.93% over this data set. © 2012 Elsevier Ltd. All rights reserved. 1. Introduction Fluorescence in situ hybridization is an established diagnostic method for gene status evaluation. It is essential in determining the status of HER-2/neu gene in breast samples, a valuable cancer prognostic and diagnostic indicator [1]. The HER-2/neu (c-erbB2) oncogene encodes the production of the HER-2/neu receptor, which is a tyrosine kinase receptor that is over-expressed in approxi- mately 20–30% of high-grade invasive breast carcinomas. Since HER-2 positive tumors can be more aggressive, knowing that a cancer is HER-2/neu positive helps in selecting the appropriate treatment. Overexpression of the protein product of HER-2/neu gene is usually a consequence of gene amplification, in which mul- tiple copies of the gene appear throughout the genome. Thus, it is possible to determine the HER-2/neu status, either by analyz- ing the numbers of gene copies in the nucleus or the amount of the related protein on the cell membrane. Fluorescence in situ hybridization (FISH) is a widely used technology to determine HER-2/neu status that allows a gene copy count. A typical FISH image of HER-2/neu is shown in Fig. 1(a). The cell nuclei have blue color, while the green and red spots map the CEP 17 and the HER- 2/neu genes, respectively. The ratio of the red/green spot numbers ∗ Corresponding author. Tel.: +30 2310996361; fax: +30 2310998453. E-mail address: imarras@aiia.csd.auth.gr (I. Marras). determines the HER-2/neu status (replication) in each cell nucleus. Alternatively, the amount of protein expression can be measured directly via immunohistochemistry (IHC). There are trade-offs in choosing one of these techniques. Both techniques permit the study of small amounts of formalin-fixed, paraffin-embedded tis- sue and the interpretation of the findings on a cell-by-cell basis. FISH allows selective staining of various DNA sequences with fluorescent markers and, thereby, the detection, analysis and quan- tification of specific numerical and structural DNA abnormalities within the nuclei. It is a direct in situ technique that is relatively rapid and sensitive. No cell culture is needed in order to apply this method and results are easier to interpret than karyotype. FISH offers a more objective scoring system, based on the presence of the two HER-2 gene signals (red/green spots) present in all cells of the specimen. Its disadvantages include the high cost of each test, the long time needed for slide scoring, the use of a fluores- cence microscope, the inability to preserve the acquired sample for long storage and review, and, occasionally the difficulty in iden- tifying the invasive tumor cells [2]. In [3,4], it has been shown that this procedure is as accurate as Southern blot analysis, while allowing the measurement of the fraction of amplified cells and the intercellular heterogeneity within a given tumor cell popula- tion. On the other hand, the advantages of IHC testing include its wide availability, relatively low cost, easy and long preservation of stained slides, while the use of specific antibodies to stain proteins in situ allows the identification of several cell types that could be 1746-8094/$ – see front matter © 2012 Elsevier Ltd. All rights reserved. http://dx.doi.org/10.1016/j.bspc.2012.05.001