2169-3536 (c) 2019 IEEE. Translations and content mining are permitted for academic research only. Personal use is also permitted, but republication/redistribution requires IEEE permission. See http://www.ieee.org/publications_standards/publications/rights/index.html for more information. This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI 10.1109/ACCESS.2019.2920383, IEEE Access 1 Date of current version 30, MAY 2019. A new hybrid image encryption algorithm based on 2D-CA, FSM-DNA rule generator, and FSBI SAJID KHAN 1* , LANSHENG HAN 2* , HONGWEI LU 3 , KHUSHBU KHALID BUTT 4 , GUEHGUIH BACHIRA 5 , AND NAIMAT-ULLAH KHAN. 6 1,4,5 School of Computer Science and Technology, Huazhong University of Science and Technology, Wuhan, China (E-mail: khansajid@hust.edu.cn) 2,3 Faculty of Computer Science Department, Huazhong University of Science and Technology, Wuhan, China 6 School of Communication and Information Engineering, Institute of Smart City, Shanghai University, China : Lansheng Han (E-mail: hanlansheng@hust.edu.cn). We are very grateful to our information security lab team members including our colleagues of the computer science department and other fellows for their valuable suggestions. We also would like to thank the anonymous referees for their valuable suggestions to improve the quality of this paper. We also would like to thanks the Natural Science Foundation of China "N0:61272033, 61572222". ABSTRACT Image encryption is an efficient and vital way to protect classified and secret images. With the advancement of the processing power of the computer, AES, DES or chaotic series type just alike image encryption schemes are not as secure as before. So in this paper, we present a new hybrid image encryption method for protecting secret and imperative images by employing Logistic Sine System (LSS) together with two- dimensional Cellular Automata and FSM based DNA rule generator. Secure hash (SHA- 256) algorithm is used to generate a secret key and to compute initial values for the LSS. In our proposed method there are three stages and each stage has its own rule. After the scrambling process, the first stage is the Feistel structure based bit inversion (FSBI) to change the pixels value. The second stage is 2D-CA with Moore neighborhood structure based local rules. The third is DNA conversion based on Finite State Machine (FSM-DNA) rule generator. The proposed encryption scheme is robust against well-known attacks such as statistical attacks, brute force attacks, differential attacks, and pixel correlation attacks and also possess strong key sensitivity. Results show that our three-layer hybrid image encryption technique is robust against many well-known attacks and can be applied directly to all type of classified gray-scale images to make them more secure from such cryptography attacks. INDEX TERMS Image encryption, DNA sequence, Finite State Machine, Feistel structure bit, Logistic Sine System, SHA-256, Two-dimensional Cellular Automata I. INTRODUCTION W ITH the fast advancement of computerized inno- vations and Internet, data security including image encryption has turned out to be increasingly critical [1]. Advanced encryption techniques inquires about has been the new hot-spot of current cryptography [2] after DES and AES turned into the standard of content information encryption. Cryptography professionals endeavor to locate an ideal im- age encryption technique to serve it as a standard image encryption algorithm. Contrasted with the content information, advanced image’s qualities as an extensive degree of information, substantial relationship, enormous information excess, and others influ- ence it to require a substantial amount of pseudo-arbitrary numbers as a key stream, which invigorates the exploration on pseudo-random numbers [3] [4]. Below, we will introduce some close related work to our paper, in order to point out the limitation of existing research work and emphasized the need for proposed research work. In [4], Logistic map based technique with permutation and image blocking proposed. In this particular technique, logis- tic map parameters and initial values functioned as the secret key. Moreover, this technique executed by the FPGA chip. This sort of permutation only images encryption algorithms have been demonstrated hazardous and was broken by chosen as well as known plain-text attack techniques [5]. Rather than utilizing the disorderly frameworks or chaotic system to produce the key-stream in [6] [7], the strategy utilized the 128-bit secret key to perform keyword dissem- ination and substitution forms. The particular encryption algorithms had certain normality and kept lake of robust- ness against the chosen plain-text attacks. The exceptional VOLUME , 2019 1