Nonsmooth Optimization for Beamforming in Cognitive Multicast Transmission A. H. Phan 1 , H.D. Tuan 1 , H.H. Kha 1 and D.T. Ngo 2 Abstract—It is well-known that the optimal beamforming problems for cognitive multicast transmission are indefinite quadratic (nonconvex) optimization programs. The conventional approach is to reformulate them as convex semi-definite programs (SDPs) with additional rank-one (nonconvex and discontinuous) constraints. The rank-one constraints are then dropped for relaxed solutions, and randomization techniques are employed for solution search. In many practical cases, this approach fails to deliver satisfactory solutions, i.e., its found solutions are very far from the optimal ones. In contrast, in this paper we cast the optimal beamforming problems as SDPs with the additional reverse convex (but continuous) constraints. An efficient algo- rithm of nonsmooth optimization is then proposed for seeking the optimal solution. Our simulation results show that the proposed approach yields almost global optimal solutions with much less computational load than the mentioned conventional one. I. INTRODUCTION A challenge for next-generation communication is spectrum shortage. Spectrum bands become apparently expensive and scarce but at the same time are still not quite effectively exploited with many spectrum holes [4]. A cognitive spec- trum share between the primary (licensed) and secondary (unlicensed) communication systems is thus an actual issue for resolution of spectrum shortage. The idea is to set addi- tional conditions to control the secondary (unlicensed) system. Namely, its communication at the frequency bands of the pri- mary (licensed) system must be cognitive enough to not disturb the latter in any way, while maintaining its own quality-of- services (QoS) requirement. Transmit beamforming has been intensively studied in the context of multiple-antenna trans- mission (e.g., MIMO systems) to exploit the spatial diversity for significantly increasing signal-to-noise-ratios (SNRs) at the receivers. Naturally, it can be used to improve the performance of cognitive transmission of the secondary systems at the frequency band of the primary system. A single multicast group scheme have been proposed in [5] and then extended to the case of multi-group [6]. Beamforming in cognitive environment in the presence of a primary system with ultimate priority has been studied in [9]. In many scenarios [3], [5], [6], [9], [10], the optimal beamforming problems are indefinite (nonconvex) quadratic optimization programs. The typical approaches for solving the problems are to exploit both the SDP relaxation and 1 School of Electrical Engineering and Telecommunications, University of New South Wales, UNSW Sydney, NSW 2052, AUSTRALIA; Email: z3261071@student.unsw.edu.au, h.d.tuan@unsw.edu.au, h.k.ha@unsw.edu.au 2 Department of Electrical and Computer Engineering, McGill University, Montreal, CANADA; Email duy.ngo@mail.mcgill.ca randomization search. Firstly, they reformulate the quadratic optimization problems as semi-definite programs (SDP) with additional matrix rank-one constraints, which are not only non- convex but discontinuous as well. These rank-one constraints are then dropped for SDP relaxations. Furthermore, the so- called randomization techniques must be employed to generate feasible solutions. In the scenario of nonzero interference on the primary system or multiple cochanel multicast groups [6], the randomization must be accompanied by an intensive number of numeric linear programs so the computational load is really heavy. A simple alternative approach for beamforming in the scenario of single group multicast has been proposed in our previous work [8]. This is a modification of the alternating projection approach [7] to directly tackle the original indefinite quadratic program. Nevertheless, our simulation [8] was able not only to show its much better performance than that of the conventional one but particularly revealed that the latter often yields solutions that are very far from the optimal ones. In this work, we propose an efficient and novel approach, which works well and consistently at any scenario of multiple cochannel multicast group. Unlike the conventional approach which drops rank-one constraints, our proposed method ex- presses the rank-one constraints as reverse convex ones. In other words, we reformulate the optimal beamforming prob- lems into SDPs with additional reverse convex (but contin- uous) constraints [13]. The optimization problems are then converted into minimization of nonsmooth (but continuous) concave function over linear matrix constraint, an important class of nonconvex optimization. An efficient iterative pro- cedure is proposed for its optimal solution. Our simulation results including comparison with lower bounds show that the approach offers an almost global optimal solution. Notations: Matrices and column vectors are denoted by boldfaced uppercase and lowercase characters, respectively. The notation A ≥ 0 means A is a (Hermitian) positive semi-definite matrix. We denote 〈A〉 = trace(A), 〈A, B〉 = trace(A H B), and 〈a, b〉 = a H b. II. OPTIMIZATION FORMULATIONS OF COGNITIVE BEAMFORMING Consider a scenario as shown in Fig. 1 where the secondary base station of N antenna elements aims at transmitting G information-bearing signals s g , g =1, ..., G to G groups G g of secondary receivers. Each group G g consists of i g secondary receivers, so the total number of the secondary receivers is M = ∑ G g=1 i g . For convenience, we use the notation i ∈G g if the secondary receiver i belongs to the 978-1-4244-5638-3/10/$26.00 ©2010 IEEE This full text paper was peer reviewed at the direction of IEEE Communications Society subject matter experts for publication in the IEEE Globecom 2010 proceedings.