Novel sensing and joint beam and null steering based resource allocation for cross-tier interference mitigation in cognitive femtocell networks Ayesha Salman 1 • Ijaz Mansoor Qureshi 1 • Shahryar Saleem 1 • Sarah Saeed 1 • Bahman R. Alyaei 1 Ó Springer Science+Business Media New York 2017 Abstract Cognitive radios and femtocell networks are gaining much popularity due to the formers ability to carry out unlicensed transmission in licensed bands and the lat- ter’s ability to increase the indoor capacity by bringing the transmitter and receiver closer to each other. The combi- nation of the two provides us with a joint benefit of unli- censed transmission and increased system capacity. However, deployments of such cognitive radio based femtocell networks (CRFN) can pose some severe cross- tier interference towards the primary system. In this paper, we propose a novel spectrum sensing technique based on Uniform Linear array (ULA) combined with Genetic Algorithm to detect not only the presence of a Primary user but also estimate its parameters such as range, frequency, angle of arrival and amplitude. Furthermore, a joint beamforming and null steering technique is proposed to maximize the signal to interference plus noise ratio (SINR) towards the secondary user equipment (SUE) while mini- mizing the interference towards the primary Base Station (PBS). We are reutilizing the sensed uplink (UL) frequency of the Primary users for the downlink (DL) transmission; the cognitive femtocell base station (CFBS) calculates such weight vector that aims at sending the maximum signal power towards the femto user and null in the direction of PBS. Keywords Cognitive radio femtocell networks  Interference mitigation  SINR maximization  Beam and null steering 1 Introduction In order to cater for the increasing demand of high speed data transmission for mobile users located indoors, Fem- tocells [1] are the proposed solution. Femtocells achieve this indoor capacity enhancement without any venture in the cellular frame [2] by bringing the transmitter and the receiver at close proximity to each other and through co- channel operation meaning re-using the same frequency that is allocated to the macrocell system and to improve indoor coverage. However, based on this co-channel operation, deployments of femtocells in underlying macrocell system can induce severe cross-tier and co-tier interference. Hence, in order to achieve maximum capacity benefit from femtocell deployments this interference should be mitigated or to some extent minimized. In [3] the authors have proposed resource allocation scheme with cross-tier interference mitigation for a non- cognitive femtocell network. Here the frequencies are being allocated by using Soft Frequency Reuse (SFR) scheme. Meanwhile, the scheme is optimized in terms of energy efficiency while guaranteeing the required data rate of both macro and femto cell users. In [4] researchers have carried out the problem of resource allocation in OFDMA- based femtocell network using a typical algorithm of & Ayesha Salman ayesha@mail.au.edu.pk Ijaz Mansoor Qureshi imqureshi@mail.au.edu.pk Shahryar Saleem shahryar.saleem@mail.au.edu.pk Sarah Saeed sarahsaeed@mail.au.edu.pk Bahman R. Alyaei bahman@mail.au.edu.pk 1 Department of Electrical and Telecom Engineering, Air University, Islamabad 44000, Pakistan 123 Wireless Netw DOI 10.1007/s11276-017-1465-6