1556 IEEE SIGNAL PROCESSING LETTERS, VOL. 22, NO. 10, OCTOBER 2015 Transceiver Optimization for Unicast/Multicast MIMO Cognitive Overlay/Underlay Networks Nikhil Gupta and Aditya K. Jagannatham Abstract—In this work, we develop a majorization theory based linear precoding framework for optimal transceiver design in MIMO cognitive radio networks. Closed form expressions are derived for the optimal MIMO precoders using two new trans- ceiver design paradigms, the zero-forcing transceiver (ZFT) and the interference optimized transceiver (IOT), for overlay and underlay MIMO cognitive radio networks respectively. Further, another novel contribution of this work is to derive the precoders for multicast MIMO cognitive radio scenarios based on novel multi-user mean-squared error (MSE) bounds. Simulation results demonstrate the performance of the proposed optimal MIMO transceivers. Index Terms—Cognitive radio, MIMO, transceiver optimiza- tion. I. INTRODUCTION C OGNITIVE radio has recently emerged as a popular new paradigm towards relieving spectral congestion [1], [2] by allowing secondary users to conditionally access spectral bands licensed to primary users. Naturally, it is essential for the secondary users to limit the interference caused to the pri- mary wireless users. Several MIMO precoding techniques have been developed to optimize the secondary transmission. Works [3], [4], [5] present beamforming algorithms for cognitive radio scenarios with single antenna users and do not focus on op- timal precoders for general multi-antenna users. Further, the proposed algorithms in [3], [4] are iterative in nature, while the work in [5] does not present any closed form solution. Optimal MIMO precoding schemes such as block diagonalization and successive optimization [6], [7] have become significantly pop- ular. The authors in [8] have developed two schemes namely the P-SVD and D-SVD for zero-forcing and interference con- strained MIMO cognitive radio transmission respectively, while the authors of [9] present a framework for secondary user rate maximization subject to a primary user rate constraint. How- ever, the above works are based on power allocation towards sum rate maximization. Realization of the sum rate performance requires a significant additional complexity in terms of forward error correction (FEC) at the transmitter and receiver, and there- fore does not correlate well with the actual BER performance. Manuscript received December 13, 2014; revised February 12, 2015; accepted March 10, 2015. Date of publication March 18, 2015; date of current version March 24, 2015. The associate editor coordinating the review of this manuscript and approving it for publication was Prof. Francesco Verde. The authors are with the Department of Electrical Engineering, Indian In- stitute of Technology, Kanpur, UP 208016, India (e-mail: nikgupta@iitk.ac.in; adityaj@iitk.ac.in). Color versions of one or more of the figures in this paper are available online at http://ieeexplore.ieee.org. Digital Object Identifier 10.1109/LSP.2015.2413940 Thus, mean squared error (MSE) minimization, which has been proposed in works such as [10], can be employed as a reliable alternative approach for performance optimization in practical cognitive radio scenarios. The work in [11] presents a frame- work for MSE minimization in cognitive radio scenarios. How- ever, the scheme presented therein is based on a iterative pro- cedure which involves repeated matrix inversion and therefore has a high computational complexity. In this context, a frame- work for optimal MIMO transceiver design towards MSE min- imization, based on majorization theory, has been presented in [10] and offers an attractive solution for optimal precoder de- sign. However, the work therein cannot be directly applied in the context of interference constrained cognitive radio scenarios. More importantly, the work therein is restricted to single user unicast scenarios. Consequently, in this letter, we propose two new MIMO transceiver design paradigms for cognitive radio networks, i.e. the zero-forcing transceiver (ZFT) and the inter- ference optimized transceiver (IOT), employing majorization theory. Depending on the cognitive radio policy, one can either choose ZFT when absolutely no secondary user interference is allowed or IOT when a marginal level of interference can be tolerated at the primary user. The proposed framework can be employed for transceiver design in diverse cognitive radio sce- narios such as overlay, where the secondary transmission causes no interference to the primary user due to intelligent signal pro- cessing and underlay, where a nominal interference level can be tolerated at the primary user. A novel contribution of this work is to present a comprehensive framework for zero-forcing and interference threshold based transceiver design for MSE minimization, whereas works such as [8], [12] consider only sum-rate optimization. We consider a variety of design objec- tives such as sum/product MSE minimization and the min-max MSE criteria to derive closed form solutions for the optimal pre- coders, which have a significantly lower computational com- plexity compared to the iterative schemes in works such as [11]. Further, for multicast scenarios arising in 3G/4G networks, pre- coder design is challenging due to the simultaneous transmis- sion to several users. In this context, we present novel precoder design expressions based on various MSE bounds derived for the sum, product and min-max MSE criteria. II. OPTIMAL MIMO TRANSCEIVERS FOR UNICAST COGNITIVE RADIO SCENARIOS Consider a cognitive radio scenario with primary users, a secondary transmitter and receiver. Let denote the number of antennas at the th primary user, while , denote the number of antennas at the secondary transmitter and re- ceiver respectively. The secondary user channel matrix is de- noted by , while the MIMO channel between 1070-9908 © 2015 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission. See http://www.ieee.org/publications_standards/publications/rights/index.html for more information.