DYNAMIC NULLING-AND-CANCELLING WITH NEAR-ML PERFORMANCE FOR MIMO COMMUNICATION SYSTEMS Dominik Seethaler, Harold Art´ es, and Franz Hlawatsch Institute of Communications and Radio-Frequency Engineering, Vienna University of Technology Gusshausstrasse 25/389, A-1040 Vienna, Austria (Europe) phone: +43 1 58801 38958, fax: +43 1 58801 38999, email: dominik.seethaler@tuwien.ac.at web: http://www.nt.tuwien.ac.at/dspgroup/time.html ABSTRACT The conventional nulling-and-cancelling (NC) detection scheme for MIMO systems uses the layerwise post-detection mean-square er- rors (MSEs) as reliability measures for layer sorting. These MSEs are average measures that do not depend on the received vector. In this paper, we propose the novel dynamic nulling-and-cancelling (DNC) detector that performs “dynamic” layer sorting based on the current received vector. Approximate a-posteriori probabilities (constructed by means of a Gaussian approximation for the post- detection interference) are used as measures of layer reliability. This results in an MMSE nulling technique that uses a simple layer- sorting rule with significantly improved performance. Our simu- lation results show that the DNC scheme can yield near-ML perfor- mance for a wide range of system sizes and signal-to-noise ratios. 1. INTRODUCTION It is well known that the nulling-and-cancelling (NC) detection scheme for MIMO systems (e.g., [1]) cannot exploit all of the avail- able diversity, and thus its performance is inferior to the perfor- mance of maximum-likelihood (ML) detection. The NC scheme uses the layerwise post-detection mean-square errors (MSEs) [2] as a reliability criterion for layer sorting. However, these MSEs are just average measures that do not depend on the received vector. Here, we propose the novel dynamic nulling-and-cancelling (DNC) detector that performs “dynamic” layer sorting based on the current received vector. At each decoding step, the DNC scheme de- tects the symbol and layer with maximum approximate a-posteriori probability (APP). The approximate APP is constructed by means of a Gaussian approximation for the post-detection interference; this approach is inspired by [3,4]. The DNC layer-sorting rule, although quite simple, can result in near-ML performance for a wide range of system sizes and signal-to-noise ratios (SNRs). We will here present the DNC scheme in a spatial multiplexing context; however, it can equally well be used for MIMO systems employing linear disper- sion codes and for multiuser detection in CDMA systems. Our paper is organized as follows. In the remainder of this sec- tion, we present the system model and briefly review existing detec- tion schemes. In Section 2, we propose and discuss the novel DNC detector. Simulation results are finally presented in Section 3. 1.1. System Model We consider a MIMO channel with M T transmit antennas and M R ≥ M T receive antennas (briefly termed an (M T , M R ) channel). We as- sume a spatial multiplexing system such as V-BLAST [1] where the ith data symbol (or layer) d i is directly transmitted on the ith trans- mit antenna. For any given time instant, this leads to the well-known baseband model r = Hd + w , (1) with the transmitted data vector d △ =(d 1 ··· d M T ) T , the M R × M T channel matrix H, the received vector r △ =(r 1 ··· r M R ) T , and the Funding by FWF grant P15156-N02. noise vector w △ =(w 1 ··· w M R ) T . The data components d i are drawn from a complex symbol alphabet A and are assumed zero-mean and independent with unit variance. The noise components w i are assumed independent and circularly symmetric complex Gaussian with variance σ 2 w . The channel H is considered constant over a block of N time instants and perfectly known at the receiver. 1.2. Review of Detection Schemes As a background and for later reference, we briefly review major detection schemes for spatial multiplexing systems. LINEAR SCHEMES. In linear equalization based schemes, the de- tected data vector is ˆ d = Q{y} with y = Gr, where G is the equalizer matrix and Q{·} denotes componentwise quantization according to ˆ d i = arg min a∈A |y i −a| 2 . (2) The zero-forcing (ZF) equalizer is given by the pseudo-inverse [5] of H. Thus, the result of ZF equalization (before quantization) is y ZF = H # r =(H H H) −1 H H r = d + ˜ w , (3) which is the data vector d plus the transformed noise ˜ w = H # w with covariance matrix R ˜ w = σ 2 w (H H H) −1 . The minimum mean-square error (MMSE) equalizer [6] minimizes the MSE E  y − d 2  and is given by G MMSE =(H H H + σ 2 w I) −1 H H , so that y MMSE =(H H H + σ 2 w I) −1 H H r . (4) NULLING- AND-CANCELLING. In contrast to linear detection, where all layers are detected jointly, NC uses a serial decision- feedback approach to detect each layer separately (e.g., [1]). At each decoding step, a single layer is detected and the corresponding con- tribution to the received vector r is then subtracted from r; the other layers that have not yet been detected are “nulled out” (equalized) using a ZF or MMSE equalizer. NC thus attempts to progressively clean r from the interference corresponding to the layers already de- tected. To minimize error propagation effects, more reliable layers should be detected first. Commonly, the layerwise post-detection MSEs are used as measures of layer reliability [2]. The resulting performance is however still significantly inferior to that of ML de- tection (see Section 3). OPTIMUM DETECTION. ML detection [7, 8] yields minimum vec- tor error probability for equally likely data vectors. For our system model (1) and our assumptions, the ML detector is given by ˆ d ML = arg min a∈A M T r−Ha 2 . The computational complexity of ML detection grows exponentially with M T . The Fincke-Phost sphere-decoding algorithm for ML de- tection [8] has an average complexity of roughly O(M 3 T ) [9]. in Proc. IEEE ICASSP-04, Montreal (Canada), May 2004, vol. IV, pp. 777–780 Copyright IEEE 2004