RESEARCH ARTICLE Provable secure lightweight hyper elliptic curve‐based communication system for wireless sensor networks Vankamamidi S. Naresh 1 | Reddi Sivaranjani 2 | Nistala V.E.S. Murthy 3 1 Department of Computer Science and Engineering, Sri Vasavi Engineering College, Tadepalligudeam, Andhra Pradesh 534101, India 2 Department of Computer Science and Engineering, Anil Neerukonda Institute of Technology and Sciences, Visakhapatnam, Andhra Pradesh 530003, India 3 Department of Mathematics, Andhra University, Visakhapatnam, Andhra Pradesh 530003, India Correspondence Vankamamidi S. Naresh, Department of Computer Science and Engineering, Sri Vasavi Engineering College, Tadepalligudeam, Andhra Pradesh 534101, India. Email: vsnaresh111@gmail.com Summary It is widely believed that hyper elliptic curve cryptosystems (HECCs) are not attractive for wireless sensor network because of their complexity compared with systems based on lower genera, especially elliptic curves. Our contribu- tion shows that for low cost security applications HECs cryptosystems can out- perform elliptic curve cryptosystems. The aim of this paper is to propose a discrete logarithm problem‐based lightweight secure communication system using HEC. We propose this for different genus curves over varied prime fields performing a full scale study of their adaptability to various types of constrained networks. Also, we propose to evaluate the performance of the pro- tocol for computational times with respect to different genus for main opera- tions like Jacobian, Divisor identifications, key generation, signature generation/verification, message encryption, and decryption by changing the size of the field. A formal security model was established based on the hardness of HEC‐Decision Diffie‐Hellman (HEC‐DDH). Finally, a comparative analysis with ECC‐based cryptosystems was made, and satisfactory results were obtained. KEYWORDS Diffie‐Hellman, elliptic curve, genus, hyper elliptic curve, Jacobian, wireless sensor networks 1 | INTRODUCTION In modern world, most of the wireless systems require resource constrained devices such as RFID tags, sensors, smart cards, small processors, PDA's, and smart phones. These devices play a major role in providing security for satellite com- munication, internet security, e‐banking, e‐commerce, Internet Of Things (IOT) applications, and embedded systems. Implementing security for wireless communication system using these devices is the most challenging problem. Many cryptographic algorithms were developed to accomplish their requirements for secure data communication in wireless systems. These algorithms have many limitations, which include increased power consumption, communication, and computational complexity with increased processing time. Thus, an efficient cryptographic algorithm that overcomes these limitations is the need of the hour. Public key cryptography (PKC) 1 offers a solution to the above limitations by using 2 different keys known as the public and private keys. The secret (private) key is chosen by the user and is well known only to him. The public key is computed from the private key by using a reversible mathematical process and is made open to all. Both the keys are interoperable on each other and are used for the decryption and encryption processes. As the private key is never revealed, PKC is highly secured unlike symmetric key cryptography. Based on the arithmetic operations, PKC is broadly Received: 5 March 2018 Revised: 11 May 2018 Accepted: 6 June 2018 DOI: 10.1002/dac.3763 Int J Commun Syst. 2018;e3763. https://doi.org/10.1002/dac.3763 © 2018 John Wiley & Sons, Ltd. wileyonlinelibrary.com/journal/dac 1 of 16