2-D DOA Estimation of Coherent Wideband Signals with Auxiliary-Vector Basis Hovannes Kulhandjian † , Michel Kulhandjian ‡ , Youngwook Kim † and Claude D’Amours ‡ † Department of Electrical Engineering, California State University, Fresno, CA 93740, USA ‡ School of Electrical Engineering and Computer Science, University of Ottawa, Canada E-mail: {hkulhandjian,youngkim}@csufresno.edu, mkk6@buffalo.edu, cdamours@uottawa.ca Abstract—We develop a two-dimensional (2-D) direction-of- arrival (DOA) estimation scheme for coherent wideband source signals using coherent signal subspace method based auxiliary- vector (CSSM-AV) basis. Computation of the basis is carried out by a modified version of the orthogonal CSSM-AV filtering algorithm. The proposed method reconstructs the signal subspace using a cross-correlation matrix after which the modified CSSM- AV algorithm is employed to estimate the azimuth and elevation angles. Then, successive orthogonal maximum cross-correlation auxiliary vectors are calculated to form a basis for the scanner- extended signal subspace. This technique is very efficient in re- ducing the algorithm complexity. Since it does not require that the eigenvectors be determined in order to find the signal subspace and yields a superior resolution performance for closely spaced sources even when the number of samples is low. Specifically, the complexity of the proposed 2-D DOA estimation algorithm compared to the CSSM algorithm is more favorable when the number of signals arriving on the antenna element is much less than the number of antenna elements. Performance evaluation shows that the proposed method outperforms competing methods such as CSSM, TOPS and WAVES algorithms in terms of estimation error, probability of resolution and number of sample support for a given SNR in scenarios in which many sources are present in the system, the array size is large, and the number of samples is small. Index Terms—Wideband direction-of-arrival (DOA) estima- tion, uniform circular arrays (UCA), auxiliary-vector (AV) fil- tering, small sample support. I. I NTRODUCTION Direction-of-arrival (DOA) estimation with sensor arrays has been a central topic of signal processing research over the past few decades due to its importance in radar, sonar, and wireless communications [1], [2], [3]. DOA estimation tech- niques can be classified into two main categories; maximum- likelihood (ML) methods, which are based on the maximiza- tion of the probability density function of the received signal and signal subspace methods that are based on the eigen- decomposition of the autocovariance matrix of the received signal. Two of the well known subspace algorithms are mul- tiple signal characterization (MUSIC) [4] and estimation of signal parameters via rotational invariance technique (ESPRIT) [5]. In general, ML-type algorithms have superior perfor- mance compared to subspace-based techniques when either the signal-to-noise (SNR) ratio or the sample size is small. Also, in the case of correlated signal sources the performance of subspace-based estimators degrades significantly, as compared to ML schemes. The trade-off of ML-type algorithms com- pared to subspace-based is the high computational complexity. These DOA algorithms are primarily designed for narrow- band signal sources and thus cannot be used for wideband signals, as the phase difference between sensor outputs is dependent on both the DOA and on the temporal frequency. A number of wideband DOA estimation algorithms are proposed, which are mainly based on coherent or noncoherent wideband techniques. The incoherent signal subspace method (ISSM) [6], [7] is one of the simplest wideband DOA estimation method. In [7], the authors propose the ISSM for wideband signals in which the received wideband signal is decomposed into a set of narrowband signals on different frequency sub- bands using the discrete Fourier transform (DFT), so that high- resolution narrowband DOA estimators such as MUSIC can be applied in each subband. In ISSM, each frequency subband is processed independently and then the separate results are averaged over all subbands. ISSM methods work well in favorable situations, i.e., high SNR and well-separated signals. Its performance may deteriorate with coherent sources or SNR variations in different frequency subbands. An outlier in any frequency subband could severely degrade the final estimate due to the averaging process. To overcome this problem a number of improved methods have been proposed. Among those are the coherent signal sub- space method (CSSM) [8]. In CSSM, the received wideband signals are first decomposed into a set of narrowband signals similar to ISSM. The covariance matrices of each subband are transformed into covariance matrices of a certain focusing frequency by multiplying them with focusing matrices. The focusing matrices are used for the alignment of the signal subspaces of narrowband components within the bandwidth of the signals, followed by the averaging of narrowband co- variance matrices into a universal covariance matrix. Then, any narrowband DOA estimators such as MUSIC can be applied to the universal covariance matrix to obtain the DOA estimates. There are a number of different methods designed for focusing matrices in the conventional CSSM. Although some of those methods are simple to implement, they require initial DOA estimates to calculate the focusing matrices. Therefore, the final DOA estimates are very sensitive to the initial estimates. The weighted average of signal subspace (WAVES) technique [9] is a widely used method, which also requires the use of 978-1-5386-4328-0/18/$31.00 ©2018 IEEE