Citation: Darooei, R.; Nazari, M.; Kafieh, R.; Rabbani, H. Dual-Tree Complex Wavelet Input Transform for Cyst Segmentation in OCT Images Based on a Deep Learning Framework. Photonics 2023, 10, 11. https://doi.org/10.3390/ photonics10010011 Received: 2 November 2022 Revised: 13 December 2022 Accepted: 15 December 2022 Published: 23 December 2022 Copyright: © 2022 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https:// creativecommons.org/licenses/by/ 4.0/). photonics hv Article Dual-Tree Complex Wavelet Input Transform for Cyst Segmentation in OCT Images Based on a Deep Learning Framework Reza Darooei 1,2 , Milad Nazari 3,4 , Rahele Kafieh 1,5, * and Hossein Rabbani 1,2, * 1 Medical Image & Signal Processing Research Center, School of Advanced Technologies in Medicine, Isfahan University of Medical Sciences, Isfahan 8174673461, Iran 2 Department of Bioelectrics and Biomedical Engineering, School of Advanced Technologies in Medicine, Isfahan University of Medical Sciences, Isfahan 8174673461, Iran 3 Department of Molecular Biology and Genetics, Aarhus University, 8000 Aarhus, Denmark 4 DANDRITE, The Danish Research Institute of Translational Neuroscience, Aarhus University, 8000 Aarhus, Denmark 5 Department of Engineering, Durham University, South Road, Durham DH1 3LE, UK * Correspondence: raheleh.kafieh@durham.ac.uk (R.K.); hoss_rab@yahoo.com (H.R.) Abstract: Optical coherence tomography (OCT) represents a non-invasive, high-resolution cross- sectional imaging modality. Macular edema is the swelling of the macular region. Segmentation of fluid or cyst regions in OCT images is essential, to provide useful information for clinicians and prevent visual impairment. However, manual segmentation of fluid regions is a time-consuming and subjective procedure. Traditional and off-the-shelf deep learning methods fail to extract the exact location of the boundaries under complicated conditions, such as with high noise levels and blurred edges. Therefore, developing a tailored automatic image segmentation method that exhibits good numerical and visual performance is essential for clinical application. The dual-tree complex wavelet transform (DTCWT) can extract rich information from different orientations of image boundaries and extract details that improve OCT fluid semantic segmentation results in difficult conditions. This paper presents a comparative study of using DTCWT subbands in the segmentation of fluids. To the best of our knowledge, no previous studies have focused on the various combinations of wavelet transforms and the role of each subband in OCT cyst segmentation. In this paper, we propose a semantic segmentation composite architecture based on a novel U-net and information from DTCWT subbands. We compare different combination schemes, to take advantage of hidden information in the subbands, and demonstrate the performance of the methods under original and noise-added conditions. Dice score, Jaccard index, and qualitative results are used to assess the performance of the subbands. The combination of subbands yielded high Dice and Jaccard values, outperforming the other methods, especially in the presence of a high level of noise. Keywords: optical coherence tomography (OCT); segmentation; fluid accumulation; deep learning; subband; dual-tree complex wavelet transform (DTCWT) 1. Introduction Diabetes is one of the fastest-growing chronic diseases, affecting more than 422 million people, especially in low- and middle-income countries [1]. Diabetic macular edema (DME) is the leading cause of blindness in the middle-aged population [2], and age-related macular degeneration (AMD), which mainly affects the elderly [3] and is an untreatable progressive condition, yields fluid leakage [4], macular cysts [5], and swelling in the central part of the retina. Optical coherence tomography (OCT) is well known in the diagnosis of DME and AMD [4,6]. Photonics 2023, 10, 11. https://doi.org/10.3390/photonics10010011 https://www.mdpi.com/journal/photonics