Modified Niederreiter type of GPT Cryptosystem based on Reducible Rank Codes Eraj Khan 1 , Ernst Gabidulin 2 , Bahram Honary 1 , and Hassan Ahmed 1 {e.khan,b.honary,h.ahmed}@lancaster.ac.uk, gab@mail.mipt.ru 1 School of Computing and Communications Infolab21, Lancaster University, UK 2 Department of Radio Engineering Moscow Institute of Physics and Technology, Russia Abstract. GPT public key cryptosystem was proposed by Gabidulin, Paramonov and Tret- jakov in 1991. This cryptosystem is based on rank error correcting codes. The main advan- tage of using rank codes in cryptography is that, it has smaller key size as compared to other code based public key cryptosystems. Several attacks against this system were published and some modifications were also proposed withstanding these attacks. In this paper, we have proposed a modified Niederreiter type GPT cryptosystem based on reducible rank codes by properly choosing the column scrambler matrix to withstand these attacks. Although, the idea of choosing column scrambler matrix from extension field is not new but the approach proposed in this paper, provides more elements of column scrambler matrix from extension field as compared to any previous modifications which makes system more secure against attacks. Keywords: GPT Cryptosystem, Rank Codes, Reducible Rank Codes, Column Scrambler 1 Introduction In 1978, McEliece proposed a public key cryptosystem based on algebraic coding theory [1]. McEliece utilized the difficulty of solving a general decoding problem. Although this system is two to three times faster than RSA and there are no successful attacks against this system but still it did not gain wide acceptance because of its very long key size and larger data expansion. Following the idea of using error correcting codes in public key cryptosystems, two other code based cryptosystems were proposed by Niederreiter [2] and Gabidulin et al. [5]. In [2], Niederreiter used the same idea proposed by McEliece but it is applied to the parity check matrix instead of generator matrix of linear codes. It used plain text as an error pattern and the resultant syndrome is the cipher text. The encryption of Niederreiter cryptosystem is about ten times faster than McEliece cryptosystem. Rank codes were introduced in [3] to correct the rank errors and public key cryptosystem based on the Maximum Rank Codes (MRD) were proposed in [4] and [5]. Compared to other two code based cryptosystems, GPT has significantly smaller key size which is mainly because of the use of rank metric instead of Hamming metric. A new family of rank codes called reducible rank codes and corresponding PKC is proposed in [6] and like Niederreiter cryptosystem, its public key is defined using parity check matrix instead of generator matrix. As rank codes are well structured therefore several structural attacks have been proposed against GPT cryptosystem. First Gibson [7], [8] and then Overbeck [9], [10] developed attacks that can completely break the GPT cryptosystem for n< 30, where n is the code length. Two approaches were proposed to withstand these attacks; advanced approach [11], [12] and smart approach [13]. In advanced approach, techniques for carefully choosing the column scrambler ma- trix P from extension field were presented whereas in smart approach technique for choosing the distortion matrix X over the extension field is proposed. Both approaches make the variants GPT