Expected Likelihood Approach for Determining Constraints in Covariance Estimation BOSUNG KANG, Member, IEEE VISHAL MONGA, Senior Member, IEEE Pennsylvania State University University Park, PA, USA MURALIDHAR RANGASWAMY, Fellow, IEEE Air Force Research Laboratory Sensors Directorate Wright-Patterson Air Force Base, OH, USA YURI ABRAMOVICH, Fellow, IEEE WR Systems Fairfax, VA, USA Recent covariance estimation methods for radar space-time adaptive processing exploit practical constraints such as the rank of clutter subspace and the condition number of disturbance covariance to estimate accurate covariance even when training is not generous. While rank and condition number are very effective constraints, often practical nonidealities make it difficult to know them precisely using physical models. Therefore, we propose a method to determine constraints in covariance estimation for radar space-time adaptive processing via an expected likelihood approach. We analyze three cases of constraints: 1) a rank constraint, 2) both rank and noise power constraints, and 3) a condition number constraint. In each case, we formulate precise constraint determination as an optimization problem. For each of the three cases, we derive new analytical results which allow for computationally efficient, practical ways of determining these constraints with formal proofs. Through experimental results from a simulation model and the KASSPER data set, we show that the estimator with optimal constraints obtained by the expected likelihood approach outperforms state-of-the-art alternatives. Manuscript received November 16, 2015; revised May 6, 2016; released for publication May 20, 2016. DOI. No. 10.1109/TAES.2016.150819. Refereeing of this contribution was handled by L. Kaplan. Research was supported by Air Force Office of Scientific Research Grant FA9550-12-1-0333. Dr. Rangaswamy was supported by the Air Force Office of Scientific Research under Project 2311IN. Authors’ addresses: B. Kang, V. Monga, Pennsylvania State University, Electrical Engineering, University Park, PA 16802; M. Rangaswamy, Air Force Research Laboratory Sensors Directorate, Radar Signal Processing, AFRL/RYRT, 2241Avionics Circle, Wright-Patterson Air Force Base, OH 45433-7132; Y. Abramovich, WR Systems, HF Radar Support, 11351 Random Hills Rd., Suite 400, Fairfax, VA 22030. Corresponding author is B. Kang, E-mail: (bkang@psu.edu). 0018-9251/16/$26.00 C  2016 IEEE I. INTRODUCTION Radar systems using multiple antenna elements and processing multiple pulses are widely used in modern radar signal processing to help overcome the directivity and resolution limits of a single sensor. Joint adaptive processing in the spatial and temporal domains for the radar systems, called space-time adaptive processing (STAP) [1–3], enables suppression of interfering signals as well as preservation of gain on the desired signal. Interference statistics—in particular the covariance matrix of the disturbance, which must be estimated from secondary training samples in practice—play a critical role in the success of STAP. Toobtain a reliable estimate of the disturbance covariance matrix, a large number of homogeneous training samples are necessary. This gives rise to a compelling challenge for radar STAP, because such generous homogeneous (target-free) training is generally not available in practice [4]. Much recent research for radar STAP has been developed to overcome this practical limitation of generous homogeneous training. Specifically, knowledge-based processing which uses a priori information about the interference environment is widely referred to in the literature [5, 6] and has merit in the regime of limited training data. These techniques include intelligent training selection [5] and reduction of spatiotemporal degrees of freedom [6–8]. In addition, covariance-matrix estimation techniques that enforce and exploit a particular structure have been pursued as one approach in these methods. Examples of structure include persymmetry [9], Toeplitz structure [10–14], circulant structure [15], and eigenstructure [16–18]. In particular, the fast maximum likelihood (FML) method [16], which enforces a special eigenstructure that the disturbance covariance matrix represents a scaled identity matrix plus a rank-deficient and positive semidefinite clutter component, falls in this category and has been shown to be the most competitive technique experimentally. Optimality claims for the FML estimator regardless of the secondary-data statistical characterization have also been provided [19]. Previous works, notably in statistics [20, 21] (and references therein), have considered factor-analysis approaches for incorporating rank information in ML estimation. Recently, Kang et al. [17] developed extensions based on convex optimization approaches and furnished closed forms for rank constrained ML (RCML) estimation in practical radar STAP. Crucially, they show that the rank of the clutter covariance, if exactly known and incorporated, enables much higher normalized signal-to-interference-and-noise ratio (SINR) and detection performance over the state of the art, particularly FML, even under limited training. Benefits of imposing low clutter rank using the knowledge-aided sensor signal processing and expert reasoning (KASSPER) data [22] was also studied in [23] for the time-varying multichannel autoregressive model, which provides an approximation to the spectral properties underlying the clutter phenomenon. IEEE TRANSACTIONS ON AEROSPACE AND ELECTRONIC SYSTEMS VOL. 52, NO. 5 OCTOBER 2016 2139