Performance Improvement Analysis of Wireless MIMO Channel in the Presence of Keyhole Md. Golam Mostafa * , Md. Masud Karim Ɨ , M. Z. Azam Ɨ , Ziaur Rahman Ɨ , Sarwar Bari Ɨ , S. P. Majumder * , Member, IEEE Ɨ Department of EECE, Military Institute of Science & Technology (MIST), Bangladesh *Department of EEE, Bangladesh University of Engineering & Technology (BUET) mostafa3593sigs@yahoo.co.uk, masudkarim521@yahoo.com, azam.1518@yahoo.com, zia_6058@yahoo.com, sarwar640@gmail.com, spmajumder2002@yahoo.com Abstract— Wireless communication systems always demand higher data rates and better quality of service. Transmission reliability in a wireless channel with high path loss, time-varying multipath fading and power and bandwidth limitations, is a testing issue. Multiple-input multiple-output (MIMO) systems, can reduce the effect of these channel interferences and achieve high-rate reliable signal transmission. Space-time block code (STBC) is a modulation scheme for the use of multiple transmits antennas providing a simple transmit diversity scheme with the same diversity order as maximal-ratio receiver combining. In certain MIMO fading environments, the offered channel capacity can be very low, where despite rich local scattering and uncorrelated transmit and receive signals, the system has only one degree of freedom. This effect has been termed as keyhole or pinhole effect. In this paper, we analyze the average symbol error rate (SER) performance of MIMO systems employing orthogonal STBC with M-PSK constellations over fading channels in the presence of the keyhole. The Nakagami-m distribution has been considered for MIMO channel modeling since a wide range of fading channels from severe to moderate, can be modeled by using Nakagami-m distribution. We derive the probability density function (PDF) of instantaneous signal-to-noise ratio and an integral equation to calculate the average SER after space–time block decoding in such channels. Numerical results show that the keyhole significantly increases the phase noise and degrades the SER performance of the STBC in MIMO channels. The performance of such channels is a function of fading figure – m and diversity. Keywords—Keyhole, MIMO channels, Nakagami-m fading, STBC, SER, PDF. I. INTRODUCTION MULTIPLE-input multiple-output (MIMO) systems, which employ multiple antennas to transmit and receive data can increase the system capacity and improve transmission reliability [1]-[3]. By transmitting multiple copies of data, a MIMO system can effectively combat the effects of fading. The Nakagami-m distribution has been considered for MIMO channel modeling since a wide range of fading channels from severe to moderate, can be modeled by using Nakagami-m distribution. In realistic MIMO fading environments, however, the existence of rank-deficient keyhole or pinhole channels can significantly reduce the diversity gain and spatial multiplexing gain of the channel. It has been demonstrated through physical experiments that MIMO systems even with uncorrelated transmit and receive signals can only have a single or reduced degree of freedom [1]-[4]. This is because that in the presence of keyhole, the channel matrix is a product of a complex Gaussian column vector and a complex Gaussian row vector and thus each entry of the channel matrix is a product of two complex Gaussian random variables. This condition can arise in a number of ways, for example, when there are two rings of scatterers at large separation. In [7], Alamouti proposed a simple transmit diversity scheme for two transmit antennas. This scheme was then generalized to an arbitrary number of transmit antennas, known as STBC. STBC is a modulation scheme for the use of multiple transmits antennas providing a simple transmit diversity scheme [7]–[9]. Due to the orthogonal structure of space–time block codes (STBCs), maximum likelihood (ML) decoding can be implemented by using the single-symbol decoding based on linear processing at the receiver. In this paper, we present the error rate analysis of orthogonal STBCs in Nakagami-m MIMO fading channels with keyhole. Furthermore, a PDF expression is derived for instantaneous SNR in order to quickly evaluate the performance of STBC in such general Nakagami-m keyhole channels, from which one can easily obtain the achievable diversity gain and coding gain performance. The rest of the paper is organized as follows. Section II presents the MIMO STBC system model. Section III describes fading channel with keyhole effect. A PDF of instantaneous SNR is derived in section IV. Our analytical results are shown in Section V. Section VI summarizes this paper. II. SYSTEM MODEL We consider a MIMO wireless communication system with   transmit and   receive antennas shown in Fig 1. The channel is assumed to be a quasi-static flat fading one with keyhole effect and the channel state information (CSI) is known at the receiver but unknown at the transmitter. Nb information bits are mapped as symbols  1 ,  2 ,….   which are selected from the MPSK