116 IEEE TRANSACTIONS ON COMMUNICATIONS, VOL. 51, NO. 1, JANUARY 2003 Beamforming, Diversity, and Interference Rejection for Multiuser Communication Over Fading Channels With a Receive Antenna Array Michael L. McCloud, Member, IEEE, Louis L. Scharf, Fellow, IEEE, and Mahesh K. Varanasi, Senior Member, IEEE Abstract—We consider -ary communication with users over a space diversity channel, consisting of a single transmit an- tenna for each user and multiple receive antennas. We examine two different flat fading models, namely, phase coherent wavefront fading and noncoherent element-to-element fading. In the case of wavefront fading, the fade is constant across the face of the re- ceive antenna and we can associate an angle of arrival to the signal. We present a variation of the MUSIC algorithm for estimating this parameter and use it to form a spatial beam. In the case of noncoherent element-to-element fading, the fading path to each sensor is different (although possibly correlated) and no angle of arrival can be exploited for conventional beamforming. For each channel model, we develop several detection strategies which as- sume various amounts of prior information about the fading. We then consider blind extensions of these detectors based on subspace tracking, which do not require a prior model for the interfering users’ signals. Index Terms—Antenna arrays, blind detection, direction of ar- rival (DOA) estimation, diversity reception, fading channels, inter- ference rejection, multiuser detection, space–time processing. I. INTRODUCTION M -ARY modulation schemes are commonly employed on noncoherent channels. The IS-95 standard, for example, employs Walsh codes on the uplink of the channel, which are decoded noncoherently. This is just one example of orthogonal multipulse modulation, and noncoherent frequency-shift keying is another. Other common techniques employ differential phase encoding and then perform detection by processing the received data two symbols at a time. The effective model employed in such detection is of an -ary constellation with each two- dimensional transmit vector corresponding to the present and previous information bit (see, e.g., [1]). In this paper, we consider several extensions of the nonco- herent multiuser detection results of [2]–[4] for -ary commu- Paper approved by A. F. Naguib, the Editor for Wireless Communication of the IEEE Communications Society. Manuscript received April 20, 2001; revised December 20, 2001. This work was supported in part by the National Science Foundation under Grant ECS-9979400 and Grant CCR-0112573, and in part by the Army Research Office under Grant DAAD 19-99-1-029. This paper was presented in part at the 35th Annual Conference on Information Sciences and Systems, Baltimore, MD, March 2001. M. L. McCloud is with Magis Networks, Inc., San Diego, CA 92130 USA (email: mccloud@ieee.org). L. L. Scharf is with the Departments of Electrical and Computer Engineering, and Statistics, Colorado State University, Ft. Collins, CO 80523-1373 USA (email: scharf@engr.colostate.edu). M. K. Varanasi is with the University of Colorado, Boulder, CO 80309-0425 USA (e-mail: varanasi@colorado.edu). Digital Object Identifier 10.1109/TCOMM.2002.807601 nication to the multiple-antenna fading channel. We consider two basic channel models. In the first case, the fading process for each user is assumed constant across the face of the array. This is called coherent wavefront fading. The second model we consider is the noncoherent element-to-element fading channel on which each sensor receives a copy of the transmitted signal with a different fading parameter. The fading coefficients may be correlated on this channel. The two channel models are de- scribed in detail in Section II. In each case, it is assumed that the fading coefficients remain constant over the duration of each symbol. They are, however, allowed to vary arbitrarily (even independently) from symbol to symbol. Such a block fading model is applicable to rapidly fading channels and/or to frequency-hopping and block-inter- leaved systems. The multiuser communication channel is fur- ther assumed to be synchronous, but this assumption may be relaxed when the blind detectors are employed. For the coherent wavefront fading channel considered in Section III, we are able to associate a direction of arrival (DOA) with each user and employ detection rules which exploit this structure. The detection schemes are extensions of the generalized maximum-likelihood (GML) and the minimum mean-squared error (MMSE) detectors presented in [2]–[4], the main difference being the inclusion of the DOA. We propose a technique for estimating the DOA which is inspired by the multiple-signal classification (MUSIC) algorithm [6]. In Section IV, we introduce two detection strategies for the noncoherent element-to-element fading channel, each appropriate for a different set of assumptions about the fading. We first develop a generalized maximum-likelihood (GML) detector under the assumption of unknown deterministic fading. This technique is also appropriate for random fading channels when the spatial statistics (correlations) of the fading process are unknown or the fading is assumed independent and identically distributed (i.i.d.) across the array. The detector acts to completely cancel multiple-access interference (MAI), making it a zero-forcing, invariant detector. The MMSE detector rule requires knowledge of the fading correlation for the user of interest. By examining the asymp- totic algebraic structure of the MMSE detector, we find a zero-forcing detector which is an analog to the multipulse decorrelating (MD) detector presented in [4]. For the case of binary signaling, the optimality of an equivalent detector was proven in [7]. The MD detector does not require knowledge of the fading correlations. 0090-6778/03$17.00 © 2003 IEEE