Optics and Lasers in Engineering 137 (2021) 106399 Contents lists available at ScienceDirect Optics and Lasers in Engineering journal homepage: www.elsevier.com/locate/optlaseng Cascaded unequal modulus decomposition in Fresnel domain based cryptosystem to enhance the image security Archana Vandana a , Sachin Sachin a,∗ , Phool Singh b a Department of Mathematics, Central University of Haryana, Mahendergarh, India-123031 b Department of Mathematics, SOET, Central University of Haryana, Mahendergarh, India-123031 a r t i c l e i n f o Keywords: Cryptanalysis Unequal modulus decomposition Fresnel transform cryptosystem Asymmetric scheme a b s t r a c t In this paper, we proposed an iterative attacking algorithm to the pure phase asymmetric optical cryptosystem based on unequal modulus decomposition. Furthermore, we have proposed a secure asymmetric cryptosystem based on cascaded unequal modulus decomposition in the Fresnel domain. Fresnel transform and unequal mod- ulus decomposition are highly sensitive to their parameters. An input grayscale image is first convoluted with a random phase mask, then Fresnel transformed, and followed by unequal modulus decomposition. This pro- cess is cascaded twice to get an encrypted image, which is quite random and looks like a noisy white image. The scheme is validated on various types of grayscale and binary images, but simulations are demonstrated for Binary dots, Boat, Jetplane, Medical, and Vegetable images. The results show that the proposed cryptosystem eliminates the silhouette problem and endures statistical and special attacks. The statistical attack based on the histogram, entropy, and correlation distribution analysis on neighbouring pixels is performed on the proposed cryptosystem. Results are shown in terms of statistical metrics such as correlation coefficient and mean squared error between plaintext and ciphertext. The proposed scheme also resists noise attack. The results show that the proposed scheme is highly secure and resist the cryptographic attacks. 1. Introduction In the era of the 21st century, fast internet has become the source of information, transmission, and storage of data. Mostly data stored or transmitted on social media remains in the form of images or videos. During storage and transmission of information, the security of data re- mains a significant issue. Data may be someone’s medical report, con- fidential military reports, government’s confidential reports, personal bank account details, online banking transaction details, student’s cre- dential of an institute, social media profile password, one-time pass- word (OTP), biometric information, national defence personals records and other sensitive information. An encryption algorithm is required to store and send data securely. Image encryption is a way to protect the information to be accessed by an unauthorized user. There are various popular digital image encryption methods such as data encryption standard (DES), advanced encryption standard (AES) available for the security of data. Still, their complex computation makes them difficult to perform in real-time. On the other hand, optical encryp- tion algorithms have features like they have less computational complic- ity, parallel processing, multidimensional parameters like wavelength, focal distance, convenient to implement, high speed, and time-saving. Optical image encryption methods attract the attention of the world af- ∗ Corresponding author. E-mail address: sachin191219@cuh.ac.in (S. Sachin). ter the double random phase encoding method introduced by Refregier and Javidi in 1995. They used two random phase masks in the spa- tial and Fourier domain and acquired a white noised type of encrypted image. Javidi highlighted and enhanced the double random phase en- coding algorithm in 1997, 2000, and 2002 [1–4]. The double random phase encoding method is used with various canonical integral domains besides the Fourier domain such as fractional Fourier, Fresnel and gy- rator transform, Hartley, fractional Hartley, fractional Mellin, wavelets, hybrid domains to develop advance image encryption algorithm [5–12]. However, later double random phase encoding is shown to be vulner- able to some basic attack like chosen plain text attack (CPA), known- plaintext attack (KPA), and ciphertext only attack (COA) [13–15]. In the light of double random phase encoding, various other symmetric optical cryptosystems are also developed [16–22]. Due to linearity prop- erty, symmetric cryptosystems are vulnerable to various cryptographic attacks [15,23–25]. To resist cryptographic attacks, various optical, dig- ital, and hybrid image encryption methods were presented with some variant of double random phase encoding. Researchers developed and enhanced asymmetric image cryptosystem based on phase reservation and phase truncation operations in the Fourier domain, known as PTFT [24,26–31]. Later on, it is found that PTFT based cryptosystems are vul- nerable to special attacks based on the iterative process [32–34]. https://doi.org/10.1016/j.optlaseng.2020.106399 Received 26 May 2020; Received in revised form 31 August 2020; Accepted 10 September 2020 0143-8166/© 2020 Elsevier Ltd. All rights reserved.