Received: 26 November 2019 Revised: 27 February 2020 Accepted: 5 March 2020 DOI: 10.1002/dac.4403 RESEARCH ARTICLE Compressed sensing channel estimation for STBC-SM based hybrid MIMO-OFDM system for visible light communication Vishwaraj B. Manur Layak Ali Department of ECE, School of Engineering, Central University of Karnataka, Kalaburagi, 585367, India Correspondence Vishwaraj B. Manur, Department of ECE, School of Engineering, Central University of Karnataka, Kalaburagi 585367, India. Email: vishwarajbm@gmail.com Summary This paper presents the idea of sparse channel estimation using compressed sensing (CS) method for space–time block coding (STBC), and spatially multiplexing (SM) derived hybrid multiple-input multiple-output (MIMO) Asymmetrically clipped optical-orthogonal frequency division multiplexing (ACO-OFDM) optical wireless communication system. This hybrid system accounts multiplexing gain of SM and diversity gain of STBC technique. We present a new variant of sparsity adaptive matching pursuit (SaMP) algorithm called dynamic step-size SaMP (DSS-SaMP) algorithm. It makes use of the inher- ent and implicit structure of SaMP, along with dynamic adaptivity of step-size feature which is compatible with the energy of the input signal, thus the name dynamic step size. Existing CS-based recovery algorithms like orthogo- nal matching pursuit, SaMP, adaptive step-size SaMP, and proposed DSS-SaMP were compared for hybrid MIMO-ACO-OFDM visible light communication system. The performance analysis is demonstrated through simulation results with respect to bit error rate, symbol error rate, mean square error, computa- tional complexity, and peak-to-average power ratio. Simulation results show that the proposed technique gives improved performance and lesser computational complexity in comparison with conventional estimation algorithms. KEYWORDS ACO-OFDM, compressed sensing, hybrid, sparse channel estimation, STBC-SM, SaMP, visible light communication 1 INTRODUCTION The substantial enormous increasing demand for data access over the wireless network has made conventional radio frequency (RF) spectrum very limited, overused, and scarcest resource. The alternative approaches like optical wireless communication (OWC) have a potential place in the next-generation wireless communication systems. 1,2 OWC has found various applications in the field of Internet of Things, security communications, ultrahigh dense networks, small-cell cov- erages, and many more. The OWC, aiding energy-efficient and cost-effective light-emitting diodes (LEDs), has promoted a most promising technology called visible light communication (VLC). The unique advantages of this technology are as follows: 3-6 1. The ultrahigh data rate for multimedia services. Int J Commun Syst. 2020;e4403. wileyonlinelibrary.com/journal/dac © 2020 John Wiley & Sons, Ltd. 1 of 19 https://doi.org/10.1002/dac.4403