Geometric Constellation Shaping Using Initialized Autoencoders Amir Omidi, Ming Zeng, Jiachuan Lin † , Leslie A. Rusch Centre for Optics, Photonics and Lasers (COPL), ECE Department, Laval University, Quebec, Canada † Huawei Technologies Canada Co., Ltd., ON, Canada amir.omidi.1@ulaval.ca, (ming.zeng, rusch)@gel.ulaval.ca, jiachuan.lin@huawei.com Abstract—Geometric constellation shaping is a promising tech- nique to boost the transmission capacity of communication sys- tems. Earlier, traditional optimization methods in constellation design lead to several advanced quadrature amplitude modulation (QAM) formats, such as star QAM, cross QAM, and hexagonal QAM. The difficulty in determining decision boundaries limited their use in real systems. To overcome this, machine learning based geometric constellation shaping has recently been proposed, where the detection is done via neural networks. Unfortunately, the resulting constellation shape is often unstable and highly dependent on initialization. In this paper, we use an autoencoder for constellation shaping and detection, with strategic initializa- tion. We contrast initialization with hexagonal QAM and square QAM. We present numerical results showing the hexagonal QAM initialization achieves the best symbol error rate performance, while the square QAM initialization has better bit error rate performance. I. I NTRODUCTION Constellation shaping is a technique to boost the trans- mission capacity of communication systems. Existing shaping methods can be broadly classified into two categories, namely geometric constellation shaping and probabilistic constellation shaping, each has its strengths and weaknesses [1], [2]. Con- sidering its relative simplicity, the focus of this paper is on geometric constellation shaping. Geometric constellation shaping optimizes the locations of the constellation points in the complex plane to achieve more compact constellations. The basic quadrature amplitude modu- lation (QAM) constellations are the well-known square QAM (SQAM) and rectangular QAM (RQAM), which are used for even and odd power of 2 constellations, respectively. With the help of geometric constellation shaping, various novel QAM constellations have been proposed, such as cross QAM (XQAM) [3], star QAM [4], hexagonal QAM (HQAM) [5]. More precisely, XQAM shifts the corner points in RQAM such that the peak and average energies of the constellation are reduced. Compared with RQAM, XQAM constellation provides at least 1 dB gain. Star QAM comprises of multiple concentric phase shift keying circles, each with an equal number of constellation points and equal phase angles. Star QAM is preferred over SQAM in systems limited by peak power. HQAM targets maximizing the minimum separation between two neighboring points, thus placing the constellation at the center of an equilateral hexagon. HQAM can either be regular or irregular (I-HQAM). Constellations of I-HQAM are set symmetric around the origin to achieve more compact packing [6], [7]. These various QAM constellations are the result of conven- tional optimization methods for geometric constellation shap- ing. Machine learning techniques for optimizing constellation shaping [8] have attracted much attention recently for this optimization. In this case, the detection can be performed straightforwardly using neural networks, without the need for calculation and application of complicated decision boundaries. Autoencoders in particular have shown great potential for con- stellation shaping. They exploit feedforward neural networks to encode binary symbols into in-phase and quadrature (I/Q) components at the transmitter and to decode I/Q components to binary symbols at the receiver [9]. Training of autoencoders is done as an end-to-end com- munication system. Data passes through two neural networks (an encoder and a decoder neural network) to reproduce the input binary stream at the output with minimum error. In each training step, the difference between transmitted data and the regenerated data is calculated and backpropagated to update the weights of both neural networks to reduce symbol error. The difference between input and output data is called training loss or training cost. The backpropagation algorithm uses a gradient descent optimization to update weights. autoencoders are explained in detail in section II. There are two types of autoencoder structures. The first aims to jointly optimize the constellation as well as the bit mapping; it has binary symbols at the input [10], [11]. However, since the bit mapping is inher- ently a discrete problem, and poorly suited to machine learning based optimization, the overall performance is unsatisfactory. The second one addresses only the constellation shaping and has one-hot data (symbols) as input. However, even in this case, shaping performance highly depends on initialization. This is most likely attributable to the constellation shaping problem being highly non-convex [12]. In this paper we use an unsupervised end-to-end learning communication system realized with autoencoders [9] to mini- mize the bit error rate (BER) and symbol error rate (SER) in an additive white Gaussian noise (AWGN) channel by geometric shaping of the constellation. We propose the use of Gray-