Real-Time Hysteresis Foreground Detection in Video Captured by Moving Cameras Hadi Ghahremannezhad Department of Computer Science New Jersey Institute of Technology Newark, NJ 07102, USA Email: hg255@njit.edu Hang Shi Innovative AI Technologies Newark, NJ 07103, USA Email: hang@iaitusa.com Chengjun Liu Department of Computer Science New Jersey Institute of Technology Newark, NJ 07102, USA Email: cliu@njit.edu Abstract—Foreground detection is an important first step in video analytics. While the stationary cameras facilitate the foreground detection due to the apparent motion between the moving foreground and the still background, the moving cameras make such a task more challenging because both the foreground and the background appear in motion in the video. To tackle this challenging problem, an innovative real-time foreground detection method is presented, that models the foreground and the background simultaneously and works for both moving and stationary cameras. In particular, first, each input video frame is partitioned into a number of blocks. Then, assuming the background takes the majority of each video frame, the iterative pyramidal implementation of the Lucas-Kanade optical flow approach is applied on the centers of the background blocks in order to estimate the global motion and compensate for the camera movements. Subsequently, each block in the background is modeled by a mixture of Gaussian distributions and a separate Gaussian mixture model is constructed for the foreground in order to enhance the classification. However, the errors in motion compensation can contaminate the foreground model with background values. The novel idea of the proposed method matches a set of background samples to their corresponding block for the most recent frames in order to avoid contaminating the foreground model with background samples. The input values that do not fit into either the statistical or the sample-based background models are used to update the foreground model. Finally, the foreground is detected by applying the Bayes classifi- cation technique to the major components in the background and foreground models, which removes the false positives caused by the hysteresis effect. Experimental evaluations demonstrate the feasibility of the proposed method in the foreground segmentation when applied to videos in public datasets. I. I NTRODUCTION Foreground segmentation has been commonly applied to intelligent surveillance systems [1]–[10]. The input video data used in the majority of these applications are captured by stationary cameras which causes the foreground to have sig- nificant motion compared to the background. A large number of studies have attempted various approaches to subtract the relatively static background from the changing foreground in order to detect the location of the moving objects [11]. However, in real-world applications camera movements are common and can happen in restricted forms, such as pan, tilt, or zoom in case of PTZ cameras used in video surveil- lance, and freely moving cameras, such as handheld cameras, smartphones, drones, or dashcams, in which case the camera is mounted on a moving platform. Consequently, there is a need to implement foreground segmentation methods that are capable of dealing with camera motion and quickly adapt to the changes in the background. The real-world applicability of the current methods for foreground detection in moving cameras suffers from high re- quirements in computational resources and/or low performance in classifying foreground and background [12], [13]. Here we apply spatial and temporal features for statistical modeling of the background and the foreground separately in order to classify them in real-time. Each block of the background is modeled using a mixture of Gaussian distributions (MOG) and a set of values sampled randomly in spatial and temporal domains. At each video frame the Lucas-Kanade optical flow method is applied on the block centers in order to estimate the camera motion and find the corresponding locations between two adjacent frames. The global motion is then compensated by updating the background models of each block according to the values of its corresponding location in the previous frame. On the other hand, the foreground is modeled by another MOG which is updated by the input values that do not fit into the background models. The final classification is performed by comparing the input super-pixel intensity values with the major components in the statistical background and foreground models. The remainder of this paper is organized as follows: In section II the main steps of the proposed framework are described in order. Section III contains experimental evaluations of the method’s performance and the conclusions are summarized in section IV. II. THE PROPOSED FOREGROUND SEGMENTATION METHOD By assuming the background to occupy the majority of the scene compared to the objects of interest we can estimate the motion of the camera relative to the background. Afterwards, the estimated camera motion can be compensated by using the corresponding values in the previous frame for updating back- ground models. Then the foreground can be segmented using approaches similar to the methods used for the applications of stationary cameras. Here, we apply an MOG to model the entire foreground using the values that are not absorbed by the