Some approach to key exchange protocol based on non-commutative groups Ruslan Skuratovskii, Aled Williams Abstract —We consider non-commutative generalization of CDH problem [1,2] on base of metacyclic group G of type Millera Moreno (minimal non-abelian group). We show that conjugacy problem in this group are intractable. The algorithm of generating (desinging) of common key in non-commutative group with 2 mutually commuting subgroups are constructed by us. Keywords — CDH and CCP problem, non-commutative cryptography, Millera Moreno group, subdirect product, generalization of CDH problem. I. INTRODUCTION In this investigation effective method of key exchange which based on non-commutative group G is proposed. The results of Ko K, Lee S, is improved and generalized [1,2,3]. Public key cryptographic schemes based on the new systems are established. One of them is most notable due to Anshel and Goldfeld [9], and another due to Ko Lee etc. As we know if CSP problem is tractable in group then problem of finding ab w by given w , 1 1 , a b - w a wa w b wb    is tractable too for arbitrary fixed w G  such that is not from center of G , where ab w is the common key that Alice and Bob have to generate. As well known if CCP problem is tractable in G then problem of finding ab w by given , w 1 1 , a b w a wa w b wb     is tractable too for arbitrary fixed w G  such that is not from center of G . Note that ab w is the common key that Alice and Bob have to generate. We denote by x w the conjugated element 1 u x wx   . We show that no eficient algorithm exists that can distinguish between the two probability distributions of ( , , ) x y xy w w w and ( , , ) g h gh w w w . Also no efficient algorithm exists to recover xh w from , w x w and . y w Metacyclic Millera Moreno group has representation 1 1 1 , , , , 2, 1, m n m P P p G ab a e eb ab a m n b          where is p prime. As a generators , ab can be chosen two arbitrary non commuting elements [4, 5,6]. The authors are with the department of computer science Kiev Polytechnic Institute, Kiev, Ukraine The second author is also with the Department of computer science of Cardiff University, UK For desining a key exchange algorithm based on non- commutative DH problem [3] it have to be effective algorithm for computation of conjugated elements. Due to the relation in metacyclic group, which define the homomphism : ( ) b Aut a   to the automorphism group of A a  , we obtaint a formula for finding a conjugated element. This formula give us possibility to efficiently calculate the conjugated to a element by using the raising to the 1 1 m p   -th power, where 1 m  . Also due to cyclic structure of groups A a  and B b  in this group G exists effectively method of cheking of equality of elements. Indead the reducing by finite modulo n give us an effective method of checking the equality of elements in the additive group n ฀ . The goal of this investigation is effective method of key exchange which based on non-commutative group G. The results of Ko K, Lee S, is improved and generalized. We consider non-commutative generalization of CDH problem [1,2] on base of metacyclic group G of Miller’s Moreno type (minimal non-abelian group). We show that conjugacy problem in this group is intractable. Effectivity of computation is provided due to using groups of residues by modulo n. The algorithm of generating (designing) common key in non-commutative group with 2 mutually commuting subgroups is constructed by us. II. PROOF THAT CONJUGACY PROPLEM IS NP-HARD IN G A. Size of cojugacy class We need to have an effective algorithm for computation of conjugated elements, if we want to design a key exchange algorithm based on non-commutative DH problem [3]. Due to the relation in metacyclic group, which define the homomorphism φ:ۦb→Aut(ۦa) to the automorphism group of the A a  , we obtain a formula for finding a conjugated element. Using this formula, we can efficiently calculate the conjugated to i a element by using the raising to the ( 1) 1 m p   -th power by modulo m p , where 1 m  . There is effective method of checking the equality of elements due to cyclic structure of subgroups A=ۦa and B=ۦb in this group G. We have an effective method of checking the equality of elements in the additive group n Z because of reducing by finite modulo n. INTERNATIONAL JOURNAL OF MATHEMATICAL MODELS AND METHODS IN APPLIED SCIENCES DOI: 10.46300/9101.2020.14.2 Volume 14, 2020 ISSN: 1998-0140 5