A Novel Approach for Rain Removal from Videos Using Low-Rank Recovery Alaa E. Abdel-Hakim Electrical Engineering Department Assiut University Assiut, Egypt, 71516 alaa.aly@eng.au.edu.eg Abstract—We propose a novel approach for rain/snow removal from videos using low-rank recovery. Rain/snow- distorted video frames are treated as a distorted 3D signal. The main goal is to separate the distortion, which is the rain or snow additive signal, from the original rain-free signal. Inter-frame information is exploited to put the problem in a convex optimization form. Then, the exact augmented La- grangian multipliers (EALM) method is used to solve the model for the low-rank terms, which represent the rain/snow-free frames. The proposed approach has several advantages over the existing approaches. It is model-independent, i.e. it does not require shape, appearance, or speed models. Also, it does not need prior information about the acquisition environment. Three different sets of data were used for evaluation: synthetic data for simulation experiments to provide quantitative results, real static videos, and real dynamic videos. The evaluation results proved the effectiveness of the proposed approach when compared to the existing approaches. Keywords-Rain removal, Low-rank recovery, RPCA, FRPCA I. I NTRODUCTION Existence of undesirable weather conditions can ruin the performance of many vision-based systems. Rain, fog, snow, or haze are big challenges for many applications, such as object detection [1]–[3], image registration [4], event detec- tion [5], attention modeling, and tracking [6]. Therefore, the removal of distortions that are caused by weather, specially rain, is growing rapidly and receiving a lot of interest. Several researchers have approached this problem in dif- ferent ways. Many of them considered model-based method- ologies. For instance, in [7], Garg and Nayar developed an approach that uses two models: one for capturing the dynamics of rain through exploiting correlation information between consequent frames. The second model is a motion- blurred-based one for modeling the rain photometry. Other approaches employ shape and/or appearance models to char- acterize the rain streaks for removal [4], [8]–[11]. Other research studies adopt early hardware adjustment strategies to reduce the effect of the rain streaks during the acquisition process itself. For example, Garg and Nayar [12] have proposed an approach to select camera parameters, e.g. the exposure time and field depth, to reduce the rain effect during at the acquisition time. However, this approach cannot deal with already-captured videos. Some recent studies, e.g. [13], attempt to solve the prob- lem for a single image by decomposing the image using morphological component analysis. Such attempts are quite useful when missing valuable temporal information that video data provides. In this paper, we solve this problem in a novel way. We propose an approach that is based on low-rank recovery. The rain/snow distortion is sparse in its nature. Thus, the rain-free scene is a low-rank when considering adjacent frames. Therefore, the proposed approach extracts the low- rank components between consecutive frames away from the sparse components. Figure 1 shows examples of the rain and snow removal using the proposed approach. The proposed approach has several advantages over the state of the art: • It is model-independent. • It has no constraints on the acquisition hardware. • All the data used to remove the rain/snow distortions is already included in the input video sequence and there is no need for prior information. II. THE PROPOSED APPROACH In this section, we explain the proposed approach. Firstly, we give a formulation of the rain removal problem from the prospective of low-rank recovery. Then, we explain the solution of the proposed model. A. Problem Statement In this work, we look at the rain/snow removal problem from a signal denoising prospective. We assume that the original video signal is a rain-free 3D signal, i.e. spatial 2D signal plus the time dimension. This noise-free signal is corrupted by some ’noise’, which is represented in the snow flakes or rain drops. The main goal is to separate the original signal from the noise components. So, the problem can be formulated as follows: The input is a video sequence that consists of N frames. The k th frame is I k ∈ R h×w , where k =1: N , h and w are the height and the width of I k , respectively. For the purpose of complying with the optimization model, let I k be rearranged in a column-vector form D k ∈ R m , m = h.w. Assume that a given frame I k is composed of two components: a low-rank rain-free component A k , and an additive sparse ’rain-noise’ component, E k . In other words,