JOURNAL OF LIGHTWAVE TECHNOLOGY, VOL. 25, NO. 1,JANUARY 2007 249 Error Probability Performance of Equi-Energy Combined Transmission of Differential Phase, Amplitude, and Polarization Moshe Nazarathy, Senior Member, IEEE, Member, OSA, and Erez Simony Abstract—A unified framework was recently developed for op- timizing the bit error rate (BER) incurred in the simultaneous differential modulation of the optical resources of polarization, phase, and/or amplitude, extending the conventional Stokes’ pa- rameters of polarization optics to a set of D 2 generalized Stokes’ parameters (GSPs). Novel optimal receiver structures were iden- tified for multienergy polarization shift keying (POLSK), the op- timality of differential phase amplitude shift keying (DPASK) and multichip differential phase shift keying (MC-DPSK) modulation formats was assessed, and optimal receivers for combinations of POLSK and DPSK were formulated. In this paper, the probability of error performance was evaluated for the newly introduced fam- ily of advanced modulation formats combining differential phase, polarization, and/or amplitude modulation, generically described as multichip differential state of POLSK. The symbol error rate and the BER for such systems are derived here in terms of the geometry of Stokes’ signal space (the space of GSPs). The resulting formalism is applied to assess the performance of recently intro- duced MC-DPSK and MUB-coded systems (differential phase con- stellations based on maximally unbiased bases), as well as DPASK formats, establishing improved tradeoffs between sensitivity and spectral efficiency relative to conventional optical DPSK systems. Index Terms—Bit error rate (BER), differential phase shift key- ing (DPSK), maximum likelihood (ML) decoding, optical commu- nication, optimal detection, pairwise error probability (PWEP), phase modulation, polarization shift keying (POLSK), state of polarization (SOP), Stokes’ parameters, symbol error rate (SER). I. I NTRODUCTION T HIS PAPER complements a recently introduced unified treatment [1] of the optical modulation formats of (dif- ferential) polarization shift keying (POLSK) [2]–[7], differen- tial phase shift keying (DPSK), differential phase amplitude shift keying (DPASK) [8]–[16], and combinations thereof, all viewed in [1] as special cases of a generic memoriless block-by- block modulation format called multichip differential state of POLSK (MC-DSOPSK), whereby it is the states of polarization (SOPs) of a block of D chips that are “multidifferentially” mod- ulated and jointly detected. Optimal receiver structures were Manuscript received September 8, 2005; revised April 19, 2006. M. Nazarathy is with the Department of Electrical Engineering, Tech- nion, Israel Institute of Technology, Haifa 32000, Israel (e-mail: nazarat@ee. technion.ac.il). E. Simony is with the Department of Neurobioloy, Weizmann Institute of Science, Rehovot 76100, Israel (e-mail: erezsim@bezeqint.net). Color versions of one or more of the figures in this paper are available online at http://ieeexplore.ieee.org. Digital Object Identifier 10.1109/JLT.2006.884211 formulated in [1] for the family of MC-DSOPSK modulation formats; however, that work stopped short of evaluating the resulting probabilities of error. It is the objective of this paper to derive the bit error rate (BER) performance for the class of optical transmission systems based on simultaneous modulation of two or all three of the optical resources of polarization, phase, and amplitude over channels wherein the optical phase slowly, yet randomly, wanders relative to the symbol rate, as applicable to high-bit-rate optical transmission. To place in perspective our current approach to BER evaluation for such systems, it is first necessary to review the concepts underlying the unified optical treatment in [1], which in turn built upon a recently developed abstract communication-theoretic framework [17], based upon two central motifs (inspired by optics, yet applicable even to nonoptical communication systems). 1) The classical description of polarization in terms of the four Stokes’ parameters was extended to D 2 general- ized Stokes’ parameters (GSPs). The associated complex two-dimensional (2-D) Jones polarization vectors were accordingly expanded to D dimensions, while the co- herency matrix of the classical polarization theory was expanded to D × D dimensions and no longer strictly describing just classical polarization but rather modeling more diverse transmission channels. 2) A generic abstract vector incoherent channel (VIC) model was identified as the underlying transmission model, theoretically accounting for block-by-block memoriless transmission over electrical and optical channels with slowly varying phase, such that the random phase is effectively constant over the transmission block of D consecutive symbols (called chips, with a chip being the smallest regular time interval over which the modulated phase/polarization/amplitude is kept constant). The VIC is suitable for modeling a large variety of wireless and optical communication systems. Building on these two key concepts, the maximum a posteriori probability (MAP) and maximum likelihood (ML) optimal detection theory over the generic VIC was formulated in [17], identifying the GSPs and extended coherency matrices as sufficient statistics. Those communication-theoretic optimal detection principles were further specifically adapted in [1] to optimize the performance of MC-DSOPSK optical transmis- sion systems, deriving optimal receiver structures based on the 0733-8724/$25.00 © 2007 IEEE