LOOSELY COUPLED LEVEL SETS FOR RETINAL LAYER SEGMENTATION IN OPTICAL COHERENCE TOMOGRAPHY Jelena Novosel 1,2 , Koenraad A. Vermeer 1 , Gijs Thepass 1 , Hans G. Lemij 3 , Lucas J. van Vliet 2 1 Rotterdam Ophthalmic Institute, Rotterdam Eye Hospital, Rotterdam, The Netherlands 2 Quantitative Imaging Group, Faculty of Applied Science, Delft University of Technology, Delft, The Netherlands 3 Glaucoma Service, Rotterdam Eye Hospital, Rotterdam, The Netherlands ABSTRACT This paper presents a novel method for the segmentation of layered structures that have a predefined order. Layers are jointly segmented by simultaneous detection of their inter- faces. This is done by means of a level set approach based on Bayesian inference where the ordering of the layers is en- forced via a novel level set coupling. The method was applied to in-vivo images of healthy human retinas acquired by opti- cal coherence tomography (OCT). A quantitative comparison with manual annotations was used to estimate the method’s accuracy, which showed very good agreement (mean absolute deviation (MAD) of 3.11-8.58 μm). The large errors were mainly due to differences in handling the vessels. Based on repeated OCT images of the same eye acquired on consecu- tive days, the reproducibility of manual and automated seg- mentations, expressed by the MAD of the RNFL thickness, were 10.97 μm and 7.68 μm. Index Terms— Level set coupling, Glaucoma, RNFL, Bayesian inference 1. INTRODUCTION Optical coherence tomography (OCT) is an in-vivo imaging technology that produces high resolution depth-resolved im- ages of the internal micro-structure of living tissue [1]. It is used in a variety of fields including ophthalmology, where high-resolution images of the retina allow diagnosis and fur- ther investigation of retinal diseases. OCT images are used to measure morphological features such as the thickness of reti- nal layers. This requires segmentation of these retinal layers which can be tedious and time consuming when done manu- ally. Therefore, a robust automatic segmentation is required. The first methods for segmentation of OCT data were based purely on processing intensity (variations) along A- lines [2], which often suffered from intensity inconsistencies within the same layer and discontinuities within layers due to shadowing artefacts resulting, for example, from blood vessels. Nowadays, segmentation methods have become more advanced, incorporating information other than inten- sity such as gradients, shape priors and other constraints. This resulted in methods for retinal layer segmentation based on active contours [3] and graph cuts [4] as well as machine learning approaches based on support vector machines [5, 6]. Although the aforementioned methods, provide reasonably good results, they do not incorporate all structural information available in segmenting the retinal layers in OCT data. Fig. 1: Cross-sectional OCT attenuation coefficient image of the retina and retinal layer definition. In this paper, a new method for segmenting layered struc- tures is presented, which is tailored to applications in oph- thalmology. The method exploits the predefined order of the retinal layers and incorporates prior knowledge about their in- tensity distributions and thickness. This method is applied to in-vivo human retinal OCT data, which was first converted to attenuation coefficients [7]. The attenuation coefficient is an optical property of the tissue and is therefore not affected by common imaging artefacts such as shading. It is therefore ex- pected to improve the robustness of all segmentation methods. In figure 1, an attenuation coefficient image of the retina is shown, including the various layer definitions. Segmentation is illustrated on the first two layers, i.e. RNFL and GCL+IPL, by detecting the interfaces that separate them. The method si- multaneously detects multiple interfaces while preserving the predefined order of the layers. Detection is done by a level set approach, where prior knowledge and image data are incorpo- rated in a probabilistic framework through the application of Bayes’ theorem. 2013 IEEE 10th International Symposium on Biomedical Imaging: From Nano to Macro San Francisco, CA, USA, April 7-11, 2013 978-1-4673-6454-6/13/$31.00 ©2013 IEEE 998