IEEE TRANSACTIONS ON BIOMEDICAL ENGINEERING, VOL. 60, NO. 10, OCTOBER 2013 2815 An Accurate Multimodal 3-D Vessel Segmentation Method Based on Brightness Variations on OCT Layers and Curvelet Domain Fundus Image Analysis Raheleh Kafieh, Student Member, IEEE, Hossein Rabbani ∗ , Member, IEEE, Fedra Hajizadeh, and Mohammadreza Ommani Abstract—This paper proposes a multimodal approach for ves- sel segmentation of macular optical coherence tomography (OCT) slices along with the fundus image. The method is comprised of two separate stages; the first step is 2-D segmentation of blood vessels in curvelet domain, enhanced by taking advantage of vessel infor- mation in crossing OCT slices (named feedback procedure), and improved by suppressing the false positives around the optic nerve head. The proposed method for vessel localization of OCT slices is also enhanced utilizing the fact that retinal nerve fiber layer becomes thicker in the presence of the blood vessels. The second stage of this method is axial localization of the vessels in OCT slices and 3-D reconstruction of the blood vessels. Twenty-four macular spectral 3-D OCT scans of 16 normal subjects were acquired us- ing a Heidelberg HRA OCT scanner. Each dataset consisted of a scanning laser ophthalmoscopy (SLO) image and limited number of OCT scans with size of 496 × 512 (namely, for a data with 19 selected OCT slices, the whole data size was 496 × 512 × 19). The method is developed with least complicated algorithms and the results show considerable improvement in accuracy of vessel segmentation over similar methods to produce a local accuracy of 0.9632 in area of SLO, covered with OCT slices, and the overall accuracy of 0.9467 in the whole SLO image. The results are also demonstrative of a direct relation between the overall accuracy and percentage of SLO coverage by OCT slices. Index Terms—Curvelet transform, fundus image, multimodal, optical coherence tomography (OCT), 3-D reconstruction, vessel segmentation. I. INTRODUCTION S PECTRAL domain optical coherence tomography (SD-OCT) is a recently established imaging technique to describe different information about the internal structures of an object and to image various aspects of biological tissues Manuscript received February 5, 2013; revised May 11, 2013; accepted May 11, 2013. Date of publication May 29, 2013; date of current version September 14, 2013. Asterisk indicates corresponding author. R. Kafieh and M. Ommani are with the Medical Image and Signal Pro- cessing Research Center, Biomedical Engineering Department, Isfahan Uni- versity of Medical Sciences, Isfahan, Iran (e-mail: r_kafieh@yahoo.com; mohammadommani@yahoo.com). ∗ H. Rabbani is with the Medical Image and Signal Processing Research Center, Biomedical Engineering Department, Isfahan University of Medi- cal Sciences, Isfahan, Iran, and also with the Iowa Institute for Biomedi- cal Imaging, The University of Iowa, Iowa City, IA 52242 USA (e-mail: h_rabbani@med.mui.ac.ir). F. Hajizadeh is with the Noor Ophthalmology Research Center, Noor Eye Hospital, Tehran, Iran (e-mail: fhajizadeh@noorvision.com). Color versions of one or more of the figures in this paper are available online at http://ieeexplore.ieee.org. Digital Object Identifier 10.1109/TBME.2013.2263844 Fig. 1. Examples on retinal 3-D OCT. (a) Three-dimensional structure with A-scans and B-scans. (b) Weak version of fundus image generated by averaging each A-scan. (c) Two-dimensional projection of the vessels produced by only averaging outer retinal layers. with micrometer resolution in depth. Three-dimensional opti- cal coherence tomography (OCT) of retina is the most popular application of this technique which can be useful in early de- tection and monitoring the progression of retinal diseases like glaucoma, diabetic retinopathy, age-related macular degenera- tion, central retinal artery (or vein) occlusion, etc. [1]–[3]. Localization of the blood vessels on retinal images may have multiple applications from being pointers of different retinal diseases [4]–[7], to playing the role of a fundamental feature in registration of retinal images of the same patient taken from different retinal areas (like optic nerve head (ONH) and macular OCTs), or taken at different visits, or even taken with different devices (like fundus images and OCTs) [8], [9]. Such a regis- tration increases the imaging area, improves the monitoring of eye disease progression, and provides more information about a specific disease, respectively. Furthermore, blood vessels can be used in comparison of two eyes (right and left eyes of one patient or the same eye during a treatment), to measure normal and abnormal features of the retina. Each retinal 3-D OCT is composed of cross-sectional scans called B-scans or transverse scans [x−z sections in Fig. 1(a)]. Such datasets contain a sizable slice of the retina, showing its internal structures in detail. Each B-scan is also comprised of successive 1-D scans in z direction in Fig. 1(a), called A-scans or axial scans. The presence of a blood vessel in retinal struc- ture causes different indicators in intersecting B-scan: a shadow appears on outer retinal layers [10], and a thickening happens in retinal nerve fiber layer (RNFL) [11]. A weak version of fundus image can be generated by averaging each A-scans [see Fig. 1(b)], with faint blood vessels due to noise and inverse effect of top retinal layers in averaging. This problem makes vessel segmentation of OCT images a challenge, in comparison to 0018-9294 © 2013 IEEE