EUROPEAN TRANSACTIONS ON TELECOMMUNICATIONS Eur. Trans. Telecomms. 2009; 20:758–769 Published online 12 March 2009 in Wiley InterScience (www.interscience.wiley.com) DOI: 10.1002/ett.1354 Optical Communications Intersymbol interference in DQPSK fibre-optic systems Hongxia Zhao 1 , Erik Agrell 1∗ and Magnus Karlsson 2 1 Department of Signals and Systems, Chalmers University of Technology, SE-412 96 G¨ oteborg, Sweden 2 Department of Microtechnology and Nanoscience, Chalmers University of Technology, SE-412 96 G¨ oteborg, Sweden SUMMARY The limited bandwidth of high-speed transmitter and receiver hardware gives rise to intersymbol interference (ISI) in digital communication systems. The influence of such ISI is investigated in a low-complexity fibre- optical setting, using differential quaternary phase-shift keying (DQPSK) modulation. The ISI is approxi- mated by a memoryless, stochastic model, for use in applications where neither equalisation nor sequence detection can be afforded. The channel capacity of this model is calculated and shown to depend strongly on the transmitter calibration. More than 3 dB is gained, at a target bit error rate (BER) of 10 −6 , by allowing the transmitted phase levels to deviate from their nominal multiples of π/2 by up to 7%. Copyright © 2009 John Wiley & Sons, Ltd. 1. INTRODUCTION In high-speed fibre-optical communication systems, the nonzero rise time of electrooptical modulators deteriorates the system performance. In combination with the receiver low-pass (LP) filters, which are also nonideal, the nonideal transmitters introduce intersymbol interference (ISI) to the demodulated signal, even at short ranges where the fibre dispersion is insignificant. In a previous paper [1], we anal- ysed the influence of nonideal hardware on coherent qua- ternary phase-shift keying (QPSK) transmission in a fibre- optical system. We showed that the ISI, if not compensated for, causes unequal bit error rates (BERs) between the two QPSK bit streams and an overall performance degradation. The work in Reference [1] is here extended to differential QPSK (DQPSK) systems. The ISI that is visible in PSK con- stellations and found to influence the two bits in the QPSK symbol differently is averaged over all transmitted phases for DQPSK, and is thus not visible as a BER difference between the two DQPSK bits. However, the effect is still present as a limitation in DQPSK systems. As will be shown, it causes unequal symbol error rates (SERs), conditioned on the transmitted symbol, and it increases the average BER. * Correspondence to: Erik Agrell, Department of Signals and Systems, Chalmers University of Technology, SE-412 96 G¨ oteborg, Sweden. E-mail: agrell@chalmers.se We also discuss and quantify how simple modifications to the transmitter can improve the system performance. From a communication-theoretical perspective, the dif- ferences between QPSK and DQPSK are significantly al- tered noise statistics and the requirement for differential precoding. From a more practical fibre communication per- spective, the DQPSK receiver is significantly easier and less complex to realise than the coherent receiver required for QPSK, although it comes at the expense of an signal-to- noise ratio (SNR) penalty. In fact, optical DQPSK systems are significantly more mature and closer to commerciali- sation than coherent QPSK systems, which is a result of having been in the research focus for the last 6–8 years. Indeed, the current transmission record over a single fibre (25 Tb/s) was enabled by DQPSK modulation [2]. It is of crucial importance for system designers, as well as telecom operators, to fully understand the cost/complexity versus performance trade-offs of DQPSK and QPSK systems. We will return to this trade-off discussion when discussing the optical transmitters in Section 2.1. DQPSK modulation is well-understood theoretically for additive Gaussian noise and ideal transmitters and receivers [3, Section 5.2.8]. It is used in numerous wireless standards Received 24 June 2008 Revised 31 December 2008 Copyright © 2009 John Wiley & Sons, Ltd. Accepted 12 January 2009