Rotated Constellations for Video Transmission over Rayleigh Fading Channels Georgia Feideropoulou, Maria Trocan, James E. Fowler, Senior Member, IEEE, B´ eatrice Pesquet-Popescu, Senior Member, IEEE, and Jean-Claude Belfiore, Member, IEEE Abstract— A joint source-channel coding scheme for transmis- sion of video over flat Rayleigh fading channels is described. The coding scheme consists of a spatiotemporal motion-compensated wavelet decomposition, a vector quantization of the coefficients through maximum-diversity lattices, and a linear labeling which minimizes simultaneously the source and channel distortion. Modulation diversity via rotated constellations produces the maximum-diversity lattices which increase robustness to channel fading without the addition of redundancy. Experimental results compare the proposed system to a prominent scalable video coder protected by more traditional convolutional codes, and superior performance is observed for high levels of channel noise. Index Terms— Joint source-channel coding (JSCC), scalable video coding I. I NTRODUCTION Video transmission over fading channels may suffer sub- stantial quality degradation due to the nature of the channel. On a fading channel, errors occur in reception when the channel attenuation is large. However, if the receiver can be supplied with several replicas of the same information signal transmitted over independently fading channels, the probability that all the signal components fade simultaneously is reduced considerably. Several strategies for such diversity reception have been developed, including frequency diversity, time diversity, and diversity techniques based on multiple antennas; an example of the latter class applied to video is [1]. In addition to these common diversity approaches, we can also speak of modulation diversity [2] wherein special multidimensional signal constellations having lattice structure are used. Such constellations provide the receiver with an order of diversity dependent on the number of dimensions of the signal constellation. The diversity order, L, of a signal set of dimension n is the minimum number of distinct components between any two constellation points. Given a Z n -lattice constellation, the desired modulation diversity is obtained by applying a suitable rotation using algebraic number-theoretical tools [2, 3]. In this letter, we consider rotated Z n -lattices with full diversity (i.e., L = n) in order to achieve reliable transmission over flat Rayleigh fading channels. G. Feideropoulou, M. Trocan, B. Pesquet-Popescu, and J.-C. Belfiore are with ´ Ecole Nationale Sup´ erieure des T´ el´ ecommunications (ENST), Paris, France. J. E. Fowler is with the Department of Electrical & Computer Engineering and the GeoResources Institute, Mississippi State University, Starkville, MS. This work was funded in part by the EC under grant IST-1-507113 (DANAE project), by the French Centre National de la Recherche Scientifique, and by the US National Science Foundation under Grant No. CCR-0310864. Previously, we have used such rotated Z n -lattices as the basis of joint source-channel coding (JSCC). In [4], it is shown that structured vector quantization (VQ) resulting from these rotated lattices combined with linear labeling simultaneously minimizes the channel and source distortions for Gaussian sources. In [5], we extended this coding scheme, originally devised for Gaussian sources, to video sequences whose source distribution is decidedly not Gaussian. Specifically, we developed a scalable video-coding system constructed from t +2D motion-compensated temporal filtering (MCTF) cou- pled with the structured VQ of [4] with several modifications to accommodate the non-Gaussianity of the spatiotemporal subbands. In [6], we coupled this JSCC video coder with partially coded index assignment for robust transmission over a binary symmetric channel, and, in [7], we paired the coder with rate-compatible punctured convolutional (RCPC) codes for the flat Rayleigh channel with independent fadings. This latter system was shown to be robust in the presence of fading; however, a trade-off was required between the compression efficiency and the added redundancy. Additionally, it was clear that the performance of a BPSK constellation without protection drops dramatically over a fading channel. In this letter, we apply modulation diversity to our JSCC video coder and demonstrate that, by rotating the given con- stellation in order to increase its diversity order, we achieve robustness to fading without adding redundancy. In addition, we compare the proposed JSCC scheme to the more traditional approach to error resilience consisting of concatenating a state- of-the-art source coder (the prominent MCTF-based coder MC-EZBC [8]) with a channel coder (RCPC codes) applied optimally for unequal error protection (UEP). Whereas per- formance of this latter MC-EZBC coder degrades quickly as channel fading increases, the proposed JSCC coder maintains performance close to that of the noiseless channel. II. JSCC FOR VIDEO A. JSCC via VQ and Linear Labeling Let a d-dimensional vector x be the input to a vector quantizer producing an n-bit binary codeword b which is the index of the vector used for signal reconstruction at the receiver. The source codebook can be viewed as a function of b =  b 1 ··· b n  T ∈{+1, -1} n = BPSK n representing a VQ index assignment. Under the assumption of a maxentropic quantizer, the total distortion is the sum of a source distortion due to quantization and a channel distortion dependent on the index assignment. In [9], it is proved that, for a binary IEEE Signal Processing Letters, vol. 14, pp. 629-632, September 2007.