Multidim Syst Sign Process DOI 10.1007/s11045-014-0290-7 2-D DOA estimation in case of unknown mutual coupling for multipath signals Tansu Filik · T. Engin Tuncer Received: 20 January 2014 / Revised: 14 July 2014 / Accepted: 17 July 2014 © Springer Science+Business Media New York 2014 Abstract In this paper, two dimensional (2-D) direction-of-arrival (DOA) estimation prob- lem in case of unknown mutual coupling and multipath signals is investigated for antenna arrays. A new technique is proposed which uses a special array structure consisting of parallel uniform linear array (PULA). PULA structure is complemented with auxiliary antennas in order to have a structured mutual coupling matrix (MCM). MCM has a symmetric banded Toeplitz structure which allows the application of the ESPRIT algorithm for 2-D paired DOA estimation. The advantage of the PULA structure is exploited by dividing it into overlap- ping linear sub-arrays (triplets) and spatial smoothing is employed to mitigate multipath signals. Closed form expressions are presented for search-free, paired and unambiguous 2-D DOA estimation. Two algorithms PULA-1 and PULA-2 are proposed to effectively solve the problem. Several simulations are done and the accuracy of the proposed solution is shown. Keywords Antenna array · 2-D direction-of-arrival estimation · Mutual coupling · Multipath · Parallel uniform linear array 1 Introduction In 2-D DOA estimation, search-free fast algorithms are usually desired for real-time prob- lems (Gershman et al. 2010; Tuncer and Friedlander 2009). The main challenge for fast 2-D DOA estimation is the pairing problem (Kikuchi et al. 2006) where multiple source azimuth and elevation angles should be paired accordingly. The proposed method in Filik and Tuncer (2010) use array interpolation techniques and virtual arrays for uniform circu- lar array (UCA), which allows efficient antenna usage and possibility to compensate the T. Filik (B ) Anadolu University, Eskisehir, Turkey e-mail: tansufilik@anadolu.edu.tr T. E. Tuncer Middle East Tecnical University, Ankara, Turkey e-mail: etuncer@metu.edu.tr 123