Lightweight Image Super-Resolution via Weighted Multi-Scale Residual Network Long Sun, Zhenbing Liu, Xiyan Sun, Licheng Liu, Rushi Lan, and Xiaonan Luo Abstract—The tradeoff between efficiency and model size of the convolutional neural network (CNN) is an essential issue for applications of CNN-based algorithms to diverse real-world tasks. Although deep learning-based methods have achieved significant improvements in image super-resolution (SR), current CNN- based techniques mainly contain massive parameters and a high computational complexity, limiting their practical applications. In this paper, we present a fast and lightweight framework, named weighted multi-scale residual network (WMRN), for a better tradeoff between SR performance and computational efficiency. With the modified residual structure, depthwise separable convolutions (DS Convs) are employed to improve convolutional operations’ efficiency. Furthermore, several weighted multi-scale residual blocks (WMRBs) are stacked to enhance the multi-scale representation capability. In the reconstruction subnetwork, a group of Conv layers are introduced to filter feature maps to reconstruct the final high-quality image. Extensive experiments were conducted to evaluate the proposed model, and the comparative results with several state-of-the-art algorithms demonstrate the effectiveness of WMRN. Index Terms—Convolutional neural network (CNN), lightweight framework, multi-scale, super-resolution.    I. Introduction I MAGE super-resolution (SR), which is to recover a visually high-resolution (HR) image from its low-resolution (LR) input, is a classical and fundamental problem in the low level vision. SR algorithms are widely used in many practical applications, like computational photography [1], scene classification [2], and object recognition [3], [4]. The SR technology can be roughly divided into two categories based on the number of input frames: multi-frame super-resolution [1], [5], [6] or single image super-resolution (SISR) [7]–[9]. In this work, we focus on SISR. Although numerous approaches have been proposed for image SR from both interpolation [10] and learning perspectives [7], [11]–[13], it is still a challenging task due to that multiple HR images can map to the same degraded observation LR. Recently, with the powerful feature representation capability, deep learning-based models have achieved superior performance in the low-level vision tasks, such as image denoising [14], [15], image deblurring [16], image deraining [17], image colorization [18] and image super-resolution [8], [19]–[28]. These methods learn a nonlinear mapping from degraded LR input to its correspon- ding visually pleasing output. Observing the advanced SISR algorithms shows a general trend that most existing convolutional neural network (CNN)- based SISR networks highly rely on increasing model depth to enhance the reconstruction performance. However, these methods would rarely deploy to solve practical problems because many devices certainly cannot provide enough computing resources. Therefore, it is crucial to design a fast and lightweight architecture to mitigate this problem [29]. To build an efficient network, we propose a weighted multi- scale residual network (WMRN) (Fig. 1) for SISR in this work. Specifically, instead of using conventional convolution operation, we first introduce depthwise separable convolutions (DS Convs) to reduce the number of model parameters and computational complexity (i.e., Multi-Adds). For exploiting and enriching multi-scale representations, we then conduct weighted multi-scale blocks (WMRBs), which adaptively filter information from different scales. By stacking several WMRBs, the representation capability can be improved. Moreover, global residual learning is adopted to add high- frequency details for reconstructing better visual results. The comparative results indicate that WMRN achieves state-of- the-art performance via a high efficiency and a small model size. In summary, the main contributions of this work include: 1) A novel weighted multi-scale residual block (WMRB) is proposed, which can not only effectively exploit multi-scale features but also dramatically reduce the computational burden. 2) A global residual shortcut is deployed, which adds high- frequency features to generate more clear details and promote gradient information propagation. 3) Extensive experiments show that the WMRN model Manuscript received September 4, 2020; revised November 4, 2020; accepted November 20, 2020. This work was supported in part by the National Natural Science Foundation of China (61772149, 61866009, 61762028, U1701267, 61702169), Guangxi Science and Technology Project (2019GXNSFFA245014, ZY20198016, AD18281079, AD18216004), the Natural Science Foundation of Hunan Province (2020JJ3014), and Guangxi Colleges and Universities Key Laboratory of Intelligent Processing of Computer Images and Graphics (GIIP202001). Recommended by Associate Editor Huimin Lu. (Corresponding author: Rushi Lan.) Citation: L. Sun, Z. B. Liu, X. Y. Sun, L. C. Liu, R. S. Lan, and X. N. Luo, “Lightweight image super-resolution via weighted multi-scale residual network,” IEEE/CAA J. Autom. Sinica, vol. 8, no. 7, pp. 1271–1280, Jul. 2021. L. Sun, Z. B. Liu, and R. S. Lan are with Guangxi Key Laboratory of Image and Graphic intelligent processing, Guilin University of Electronic Technology, Guilin 541004, China (e-mail: cs.longsun@gmail.com; 3936924@ qq.com; rslan2016@163.com). X. Y. Sun and X. N. Luo are with National Local Joint Engineering Research Center of Satellite Navigation and Location Service, Guilin University of Electronic Technology, Guilin 541004, China (e-mail: 173610744@qq.com; luoxn@guet.edu.cn). L. C. Liu is with the College of Electrical and Information Engineering, Hunan University, Changsha 410082, China (e-mail: rslan2016@163.com). Color versions of one or more of the figures in this paper are available online at http://ieeexplore.ieee.org. Digital Object Identifier 10.1109/JAS.2021.1004009 IEEE/CAA JOURNAL OF AUTOMATICA SINICA, VOL. 8, NO. 7, JULY 2021 1271