International Journal of Wireless & Mobile Networks (IJWMN) Vol. 3, No. 4, August 2011 DOI : 10.5121/ijwmn.2011.3414 205                  Rim Haddad 1 , Ridha Bouallègue 2 Laboratory Research in Telecommunication 6’Tel in High School of Communication of Tunis, Route de Raoued Km 3.5, 2083 Ariana, Tunisia 1 rim.haddad@yahoo.ca 2 ridha.bouallegue@supcom.rnu.tn ABSTRACT The main role of smart antennasis to mitigate Multiple Access Interference (MAI) by beamforming (Spatial filtering) operation. In addition to MAI, the performance of receivers is limited by fast fading. In this context, we propose in this paper a hybrid scheme of beamforming and diversity called HBF (Hierarchical Beamforming) and we propose a system model for the mathematical characterization of HBF for the performance evaluation.Moreover, we compare the performance of HBF receiver with conventional Beamforming (CBF) one. The proposed model conforms the benefits of adaptive antennas in reducing the overall interference level (intercell/intracell) and to find an accurate approximation of the error probability. KEYWORDS Beamforming, Hierarchical Beamforming (HBF), Conventional Beamforming (CBF), BER, Rayleigh fading 1. INTRODUCTION As the growing demand for mobile communications is constantly increasing, the need for better coverage, improved capacity and higher transmission quality rises. Thus, a more efficient use of the radio spectrum is required. Smart antenna systems have emerged as one of the most efficiency and improving the performance of present and future wireless communication systems. Several smart antenna systems have been proposed and demonstrated at the base station (BS) of the wireless communication system, and these have shown that significant increases in capacity are possible. The deployment of smart antennas at cellular base station installations has gained enormous interest because it has the potential to increase cellular system capacity, extend radio coverage, and improve quality of services [1,2]. In a typical mobile environment, signals from users arrive at different angles to the base station and hence antenna arrays can be used to an advantage. Each multipath of a user may arrive at a different angle, and this angle spread can be exploited using an antenna array [3]. The Bit Error Rate (BER) is considered to be one of the most important performance measures for communication systems and hence it has been extensively studied. In our paper, we propose a novel approach to evaluate the average probability of error by considering an approximation of the spatial filter. Hence, we will derive an analytical model for