Comparative investigations of DWDM transmission system with PMD for different orthogonally modulated signals at 80 Gb/s Richa Bhatia 1, * ,† , Ajay Kumar Sharma 2 and Jyoti Saxena 3 1 Department of Electronics and Communication Engineering, Ambedkar Institute of Advanced Communication Technologies and Research, Delhi, India 2 Department of Computer Science and Engineering, National Institute of Technology, Delhi, India 3 Department of Electronics and Communication Engineering, Giani Zail Singh PTU Campus, Bathinda, Punjab, India SUMMARY Polarization mode dispersion (PMD) has become one of the major limiting factors for high-bit-rate optical trans- mission systems. This paper evaluates the performance of dense wavelength division multiplexed (DWDM) sys- tem with PMD at 80 Gb/s in the presence of Kerr-nonlinear effects. Orthogonally modulated signals have been investigated and compared for tolerance against PMD in a DWDM transmission system with direct detection receivers. The optimized combinations of orthogonal polarization (OP) with carrier-suppressed return-to-zero (CSRZ) and CSRZ differential-phase-shift-keying signals are shown to improve PMD tolerance over high bit rates and long transmission lengths. Improved performance greater than 4 dB is observed for CSRZ modu- lated signal with OP because of less channel cross talk and reduced power transfer between adjacent bits over different PMD values. The numerical results demonstrate that our proposed orthogonally modulated signals perform better with lesser complex direct detection receivers. Copyright © 2015 John Wiley & Sons, Ltd. Received 26 April 2015; Revised 19 July 2015; Accepted 1 September 2015 KEY WORDS: DWDM system; carrier-suppressed return-to-zero; orthogonal polarization system; high bit rate; polarization mode dispersion 1. INTRODUCTION To achieve high per-channel bit rate and high bandwidth efficiency in modern optical communica- tion system, the choice of a suitable modulation format is still an important issue. There is a tremen- dous pressure to investigate novel modulation formats at lower complexity and cost, for designing next generation optical systems carrying high data rate for each user [1–3]. Lyubomirsky and Chien [4] proposed a lower complexity differential-phase-shift-keying (DPSK) receiver based on a frequency discriminator demodulator with direct detection (DD) and proved it to be less sensitive to non-optimum frequency offsets, with better chromatic dispersion tolerance, increased tolerance to nonlinear Gordon–Mollenauer effect, and the filter method of DD also holds a significant amplified-spontaneous-emission noise rejection advantage [5–7]. Hence, the excellent dispersion and nonlinear tolerance of DD scheme along with its lower complexity receiver structure makes it an interesting alternative for future high-bit-rate transport systems. However, intensity noise resulting from the phase modulation to intensity modulation conversion of laser phase noise can be a major impairment in DD systems, and fiber nonlinearity can significantly enhance the relative intensity noise (RIN) [8]. This RIN can be reduced by suitable choice of system parameters and appropriate choice of narrow linewidth laser source [9, 10]. *Correspondence to: Richa Bhatia, Department of Electronics and Communication Engineering, Ambedkar Institute of Advanced Communication Technologies and Research, Delhi, India. † E-mail: richatimes@gmail.com Copyright © 2015 John Wiley & Sons, Ltd. INTERNATIONAL JOURNAL OF COMMUNICATION SYSTEMS Int. J. Commun. Syst. 2016; 29:1084–1090 Published online 5 October 2015 in Wiley Online Library (wileyonlinelibrary.com). DOI: 10.1002/dac.3076