2804 IEEE TRANSACTIONS ON VEHICULAR TECHNOLOGY, VOL. 57, NO. 5, SEPTEMBER 2008 Extending a Fixed-Complexity Sphere Decoder to Obtain Likelihood Information for Turbo-MIMO Systems Luis G. Barbero, Member, IEEE, and John S. Thompson, Member, IEEE Abstract—A list extension for a fixed-complexity sphere de- coder (FSD) to perform iterative detection and decoding in turbo- multiple input-multiple output (MIMO) systems is proposed in this paper. The algorithm obtains a list of candidates that can be used to calculate likelihood information about the transmitted bits required by the outer decoder. The list FSD (LFSD) overcomes the two main problems of the list sphere decoder (LSD), namely, its variable complexity and the sequential nature of its tree search. It combines a search through a very small subset of the complete transmit constellation and a specific channel matrix ordering to approximate the soft-quality of the list of candidates obtained by the LSD. A simple method is proposed to generate that subset, extending the subset searched by the original FSD. Simulation re- sults show that the LFSD can be used to approach the performance of the LSD while having a lower and fixed complexity, making the algorithm suitable for hardware implementation. Index Terms—Iterative decoding, list sphere decoder (LSD), multiple input-multiple output (MIMO), turbo decoding, wireless communications. I. I NTRODUCTION T HE USE of multiple-input-multiple output (MIMO) tech- nology has become the new frontier of wireless com- munications after theoretical analysis showed that significant capacity increases could be achieved under certain conditions by using multiple antennas at both transmitter and receiver [1]. This technology can potentially improve the reliability or speed of current and future wireless systems like wireless local are networks (WLAN) or fourth generation cellular systems (4G). In particular, it has been shown that the capacity of the channel can be approached using a turbo-MIMO scheme based on bit-interleaved coded modulation (BICM) [2]. This scheme consists of a combination of a spatially-multiplexed MIMO stage and an outer code with an interleaver operation in between [3], [4]. Manuscript received August 31, 2006; revised July 13, 2007, September 13, 2007, and November 15, 2007. This work was supported by Alpha Data Ltd. and the Scottish Funding Council through the Edinburgh Research Partnership. The review of this paper was coordinated by Prof. M. Juntti. L. G. Barbero was with the Institute for Digital Communications, Joint Research Institute for Signal & Image Processing at the University of Edinburgh, EH9 3JL Edinburgh, U.K. He is now with the Institute of Electron- ics, Communications and Information Technology at the Queen’s University of Belfast, BT3 9DT Belfast, U.K. (e-mail: l.barbero@ecit.qub.ac.uk). J. S. Thompson is with the Institute for Digital Communications, Joint Re- search Institute for Signal & Image Processing at the University of Edinburgh, EH9 3JL Edinburgh, U.K. (e-mail: john.thompson@ed.ac.uk). Digital Object Identifier 10.1109/TVT.2008.914064 In such a system, the turbo-principle can be applied between the soft-MIMO detector and the outer decoder performing iterative detection and decoding [5]. The list sphere decoder (LSD) is considered the most promising algorithm for soft- MIMO detection, reducing the high complexity of the maxi- mum likelihood detector (MLD), especially for large number of antennas or constellation orders [2]. Given that the LSD is an extension of the sphere decoder (SD) proposed for uncoded MIMO detection [6], it suffers from the same disadvantages: a variable complexity, that depends on the channel conditions and the noise level, and a sequential tree search [7]. Those two aspects affect a possible hardware implementation of the algorithm, resulting in a variable throughput and a suboptimum use of the hardware resources [8], [9]. Since the introduction of the LSD using the Fincke-Pohst (FP) enumeration [2], different alternatives have been proposed to improve its performance and, in some cases, reduce its com- plexity. However, most of them still have a variable complexity and perform a sequential search. They can be classified in the following categories: • Use of the a priori information of the bits to improve the soft-quality of the list of candidates in every iteration [10], [11]. In these cases, the soft-MIMO detector is run in each iteration significantly increasing the final complexity of the receiver. • Reduction of the complexity of the LSD [12] using the Schnorr-Euchner (SE) enumeration [13]. • Use of the SE enumeration together with additional oper- ations to improve the list of candidates for iterative detec- tion and decoding [14], [15]. Although the SE enumeration is used, the additional operations can cause an increase in the overall complexity of the algorithm. • Application of the M-algorithm (i.e. K-Best lattice de- coder) [16]. This approach provides a fixed complexity that, in most cases, is higher than that of the LSD. Al- ternatives to reduce its complexity have been proposed, although the fixed complexity is no longer achieved [17], [18]. In this paper, a list version of a fixed-complexity sphere decoder (FSD) is proposed. The FSD algorithm has been pre- viously proposed to achieve quasi-maximum likelihood (ML) performance in uncoded MIMO detection, combining a fixed search through the transmit constellation and a novel channel matrix ordering [19], [20]. The list FSD (LFSD) presented here performs an extended search compared to that of the FSD using 0018-9545/$25.00 © 2008 IEEE