RESEARCH ARTICLE Influence of nonlinear effects on 6.4 Tb/s dual polarization quadrature phase shift keying modulated dense wavelength division multiplexed system Djima Kassegne 1 | Simranjit Singh 2 | S. Sanoussi Ouro‐Djobo 1 | Barerem‐Melgueba Mao 1,3 1 Physics Department, University of Lomé, BP 1515 Lomé, Togo 2 ECE Department, Punjabi University, Patiala, Punjab 147002, India 3 CIC, University of Lomé, BP 1515 Lomé, Togo Correspondence Djima Kassegne, Physics Department, University of Lomé, BP 1515 Lomé, Togo. Email: djimakassegne@gmail.com Summary In this paper, we have proposed a high‐performance transmission system of 32 channels based on dense wavelength division multiplexing (DWDM) through an optical fiber link consisting of 110 km per span. To check the influence of nonlinear effects, we have considered phase modulation (SPM), cross‐phase modulation (XPM), four‐wave mixing (FWM), stimulated Raman scattering (SRS), and self‐steepening (SS) which limits the performance of optical commu- nication systems. In the proposed system, each channel is modulated at 200 Gbps of dual‐polarization quadrature phase shift keying (DP‐QPSK) format, with a 50‐GHz grid yield whole data rate of 6.4 Tbps. The results have proved that the nonlinear phenomena degrade the performance of proposed optical fiber communication systems, and XPM has the most dominant effect, followed by SS, SRS, and SPM. The simulation was carried out with Optisystem 14 which is powerful software for modeling and simulation of optical fiber transmission. KEYWORDS DWDM system, modulation format, nonlinear effects 1 | INTRODUCTION Optical fiber remains the backbone of communication systems because of its enormous advantages, including low cost, large bandwidth, and compatibility with new multiplexing techniques. Especially, the use of wavelength division multiplexing (WDM) technology in optical fiber transmission would be a promising solution to meet the growing demand for bandwidth. 1,2 Since the discovery of the laser in the 1960s, light sources used in optical fiber transmission systems are very intense. 3,4 This could lead to interactions, either between the signal and the transmission medium (optical fiber), or between the signals themselves in the case of the simultaneous transmission of several signals via a single optical fiber. One of the effects resulting from the interactions in optical fiber is the dependence of the refractive index of the fiber on the signal intensity 2,5,6 ; this is known as the optical “Kerr effect” and is involved in optical transmissions nonlinear phenomena such as self‐phase modulation (SPM), cross‐phase modulation (XPM), and four‐wave mixing (FWM). 5,7-10 The interactions between the signals and the optical fiber can also lead to an exchange of energy between the fiber and the signals when signal power becomes very intense. This introduces the second category of nonlinear effects, which appear for ultra‐short pulses (less than 0.1 ps): these are stimulated Raman scattering (SRS) and self‐steepening (SS) phenomena. 9-12 Even if some of nonlinear effects (under certain conditions) could offer a variety of possibilities (for ultra‐fast all‐ optical switching, amplification, and regeneration), 9 several works have shown that nonlinear effects limit the capacity Received: 2 February 2019 Revised: 23 April 2019 Accepted: 30 April 2019 DOI: 10.1002/dac.4021 Int J Commun Syst. 2019;e4021. https://doi.org/10.1002/dac.4021 © 2019 John Wiley & Sons, Ltd. wileyonlinelibrary.com/journal/dac 1 of 12