American Journal of Applied Sciences 9 (9): 1443-1447, 2012 ISSN 1546-9239 © 2012 Science Publication Corresponding Author: E.S. Ismail, School of Mathematical Sciences, Faculty of Science and Technology, Universiti Kebangsaan Malaysia, 43600 UKM Bangi, Selangor, Malaysia 1443 Development of a New Elliptic Curve Cryptosystem with Factoring Problem E.S. Ismail and M.S. Hijazi School of Mathematical Sciences, Faculty of Science and Technology, Universiti Kebangsaan Malaysia, 43600 UKM Bangi, Selangor, Malaysia Abstract: Problem statement: The security of elliptic curve cryptosystems are based on elliptic curve discrete logarithm problem (ECDLP). However, if an attacker finds a solution to ECDLP, the elliptic curve-based systems will no longer be secure. Approach: To improve this, we develop a new elliptic curve cryptosystem using one of the old/novel problem in computational number theory; factoring problem (FAC). Specifically, our encrypting and decrypting equations will heavily depends on two public keys and two secret keys respectively. Results: We show that, the newly designed cryptosystem is heuristically secure against various algebraic attacks. The complexity of the scheme shows that the time complexity for each encryption and decryption are given by 299T mul and 270T mul . Conclusion: The new system provides greater security than that system based on a single hard problem. The attacker has not enough resources to solve the two hard problems simultaneously in a polynomial time. Key words: Cryptosystem, elliptic curve, factoring problem, elliptic curve discrete logarithm problem INTRODUCTION Diffie and Hellman (1976) were the first to propose the idea of transmitting secret message between two communicating parties; a sender and a receiver in an insecure channel (with the presence of attackers). Their idea (is called cryptosystem) consists of these following properties: • The sender first encrypts the message using receiver’s public key and sends the encrypted message to the receiver • The receiver who possesses the secret key can decrypt and read the original message • The security of the system is depends on the underlying hard problems in computational number theory • Knowing only the public key of receiver, the attacker is not able to read the message since he has no information about the corresponding secret key Unfortunately, they did not develop any such system. The first realization was developed by Rivest et al. (1978) and is called RSA cryptosystem after their first names. The security of RSA is based on the hardness of solving factoring problem (FAC). Informally, if the attacker manages to solve FAC, the underlying system will no longer be secure. With the proper selection of parameters, no one is able to break the novel RSA system. Rabin (1979) designed a new cryptosystem whose security is depends heavily on residuosity problem (RES). His system relies on the difficulty of finding prime divisors of a given large composite integer as in RSA. However, no concrete relationship between the hardness of solving FAC and RES is found. Six years later, Elgamal (1985) proposed his new cryptosystem based on Discrete Logarithm Problem (DLP). Later, Koblitz (1987) and Miller (1986) independently proposed the use of elliptic curve in cryptosystems. Their security lies on the so-called Elliptic Curve Discrete Logarithm Problem (ECDLP). Their systems are more efficient than previous systems since the size of the main parameter is only 160-bits. Many such systems were then been developed (Menezes, 1993; Rabah, 2005). One common feature of these schemes is that the security of the systems is based on a single hard problem. If one day in a near future an attacker solves the hard problem, the underlying system will no longer be secure. Thus to overcome this disadvantage, many designers are proposing cryptosystems based on two hard problems (Baocang and Yupu, 2005; Elkamchouchi et al., 2004; Harn, 1994; Ismail and Hijazi, 2011). If the attacker find a solution to one of these hard problem the system stays secure as the another problem is still hard to solve. It is impossible for the attacker to solve the two problem simultaneosly. In this study, we develop a new cryptosystem based on two hard problems; ECDLP and FAC. A desirable system with two hard problems should come with the following criteria: (1) the system uses only one pair of public and private keys; (2) each