International Journal of Electrical and Computer Engineering (IJECE) Vol. 13, No. 4, August 2023, pp. 4640~4648 ISSN: 2088-8708, DOI: 10.11591/ijece.v13i4.pp4640-4648  4640 Journal homepage: http://ijece.iaescore.com Deep learning-based switchable network for in-loop filtering in high efficiency video coding Helen K. Joy 1 , Manjunath R Kounte 2 1 School of Electronics and Communication Engineering, REVA University, Bengaluru, India 2 Department of Electronics and Computer Engineering, School of Electronics and Communication Engineering, REVA University, Bengaluru, India Article Info ABSTRACT Article history: Received Sep 6, 2022 Revised Dec 11, 2022 Accepted Dec 21, 2022 The video codecs are focusing on a smart transition in this era. A future area of research that has not yet been fully investigated is the effect of deep learning on video compression. The paper’s goal is to reduce the ringing and artifacts that loop filtering causes when high-efficiency video compression is used. Even though there is a lot of research being done to lessen this effect, there are still many improvements that can be made. In This paper we have focused on an intelligent solution for improvising in-loop filtering in high efficiency video coding (HEVC) using a deep convolutional neural network (CNN). The paper proposes the design and implementation of deep CNN-based loop filtering using a series of 15 CNN networks followed by a combine and squeeze network that improves feature extraction. The resultant output is free from double enhancement and the peak signal-to-noise ratio is improved by 0.5 dB compared to existing techniques. The experiments then demonstrate that improving the coding efficiency by pipelining this network to the current network and using it for higher quantization parameters (QP) is more effective than using it separately. Coding efficiency is improved by an average of 8.3% with the switching based deep CNN in-loop filtering. Keywords: Coding tree unit Convolutional neural network Deep learning High efficiency video coding In-loop filtering Video coding This is an open access article under the CC BY-SA license. Corresponding Author: Helen K. Joy School of Electronics and Communication Engineering, REVA University Bengaluru, 560064, India Email: helenjoy88@gmail.com 1. INTRODUCTION In recent years, as the usage of video, is in high demand, the advancement needed in video codec is also highly prioritized. The transition of video codec to a hybrid model and smart video codec is the trend of research now. As the codec mostly follows block-based coding the chances of ringing effect and visible block structure are high. The in-loop filter in high efficiency video coding (HEVC) is used to alleviate this effect. The compression artifact reduction is a matter of concern in this. HEVC specifies two in-loop filters, deblocking and sample adaptive offset (SAO), which considerably increase the subjective quality of decoded video sequences as well as compression efficiency by improving the quality of the reconstructed images. The SAO primarily corrects ringing artifacts produced by massive transformations and quantization, as well as sample value offsets in specific sections of an image generated by coding of motion vectors, while the deblocking filter reduces discontinuities on the block borders. Optionally, one or two of these filtering processes can be applied before placing the reconstructed image in the decoded picture buffer (DPB). The de- blocking filter (DBF) is employed in the same way as in H264/ advanced video coding (AVC) [1], although the DBF has been simplified in terms of decision making and filtering. SAO is a nonlinear amplitude mapping filter that works with DBF data. The purpose of SAO is to improve signal amplitude reconstruction