JOURNAL OF LIGHTWAVE TECHNOLOGY, VOL. 29, NO. 4, FEBRUARY 15, 2011 571 64-Tb/s, 8 b/s/Hz, PDM-36QAM Transmission Over 320 km Using Both Pre- and Post-Transmission Digital Signal Processing Xiang Zhou, Senior Member, IEEE, Jianjun Yu, Senior Member, IEEE, Ming-Fang Huang, Yin Shao, Ting Wang, Lynn Nelson, Peter Magill, Martin Birk, Peter I. Borel, David W. Peckham, Robert Lingle, Jr., and Benyuan Zhu Abstract—We report the successful transmission of 64 Tb/s capacity (640 107 Gb/s with 12.5 GHz channel spacing) over 320 km reach utilizing 8-THz of spectrum in the -bands at a net spectral efficiency of 8 bit/s/Hz. Such a result is accomplished by the use of raised-cosine pulse-shaped PDM-36QAM modula- tion, intradyne detection, both pre- and post-transmission digital equalization, and ultra-large-area fiber. We discuss in detail the digital modulation technology and signal processing algorithms used in the experiment, including a new two-stage, blind fre- quency-search-based frequency-offset estimation algorithm and a more computationally efficient carrier-phase recovery algorithm. Index Terms—Capacity, coherent, digital, fiber, modulation format, optical transmission, QAM, spectral efficiency. I. INTRODUCTION I NCREASING the spectral efficiency has historically been shown to be an effective method to lower the cost per transmitted bit because more capacity can be shared by common infrastructure such as optical amplifiers and fiber. Recent progress in digital coherent detection technology cou- pled with the use of advanced multi-level, multi-dimensional modulation formats has resulted in significant improvement of spectral efficiency [1]–[13] and, therefore, the overall fiber capacity [1]–[6]. For example, by using return-to-zero (RZ) shaped polarization-division-multiplexed (PDM) 8-ary phase shift keying (PSK) and digital coherent detection, 17 Tb/s ca- pacity at 114 Gb/s per channel data rate and 4 bit/s/Hz spectral efficiency has been transmitted over 660 km reach using only the C-band EDFA bandwidth [1]. With more noise-tolerant PDM-RZ-8QAM (quadrature amplitude modulation) signals, 34 Tb/s net capacity over 580 km reach has been reported within the standard 8 THz -band [3]. By using the 10.8-THz-wide C- and extended L-bands along with PDM-16QAM, 69.1 Tb/s capacity has been transmitted over 240 km at a spectral effi- ciency of 6.4 bit/s/Hz and per channel data rate 171 Gb/s [6]. Manuscript received June 21, 2010; revised November 04, 2010, December 23, 2010; accepted January 04, 2011. Date of publication January 20, 2011; date of current version February 08, 2011. X. Zhou, P. D. Magill, L. Nelson, and M. Birk are with AT&T Labs-Research, Middletown, NJ 07748 USA (e-mail: zhoux@research.att.com). J. Yu was with NEC Laboratories America, Inc., Princeton, NJ 08540. He is currently with ZTE USA Inc., Iselin, NJ 08830 USA. M. Huang, Y. Shao, T. Wang, are with NEC Laboratories America, Inc., Princeton, NJ 08540. P. I. Borel, D. W. Peckham, R. Lingle, Jr., and B. Zhu are with OFS, Norcross, GA 30071 USA. Color versions of one or more of the figures in this paper are available online at http://ieeexplore.ieee.org. Digital Object Identifier 10.1109/JLT.2011.2105856 TABLE I OVERVIEW OF RECENTLY ACHIEVED HIGH-SE WDM SYSTEMS In order to further increase the fiber capacity, higher spectral efficiency has also been explored by using even higher order modulation formats. In Table I we give an overview of recent high-spectral-efficiency WDM research demonstrations. So far the highest spectral efficiency demonstrated in a full DWDM transmission experiment is 9 b/s/Hz, reported in [9] with a total capacity of 11.2 Tb/s and a transmission reach of 160 km. In this paper we describe our recent transmission experiment of 64 Tb/s capacity at 8 bit/s/Hz over 320 km reach. This exper- iment used raised-cosine pulse-shaped PDM-36QAM modula- tion, intradyne detection, and both pre- and post-transmission digital equalization. Some of the experimental results have been presented in [5]. This paper presents a more detailed discussion of the utilized technologies as well as the measured results. The remainder of this paper is organized as follows. In Section II we give a detailed description of the experimental setup, including the digital modulator technology, line system, and the receiver configuration. Section III is devoted to the dig- ital signal processing (DSP) algorithms implemented at the co- herent receiver, where we put special emphasis on the frequency and phase recovery algorithm. The measured back-to-back and transmission results are discussed in Section IV. Finally we summarize the paper in Section V. II. EXPERIMENTAL SETUP The experimental setup is shown in Fig. 1(a)–(c). Fig. 1(a) shows the transmitter setup. It includes two transmitters, one for the 320 -band odd wavelength channels and the other 0733-8724/$26.00 © 2011 IEEE