Bi-Layer Video Segmentation with Foreground and Background Infrared Illumination Qiong Wu M.Sc. Student University of Alberta 1-780-695-7953 qiong@cs.ualberta.ca Pierre Boulanger Professor University of Alberta 1-780-492-3031 pierreb@cs.ualberta.ca Walter F. Bischof Professor University of Alberta 1-780-492-3114 wfb@cs.ualberta.ca ABSTRACT In this paper, we investigate two ways of employing infrared video with color video for automatic foreground-background video segmentation: foreground infrared (IR) illumination and background IR illumination. Foreground IR illumination gives an initial foreground template, which is combined with image segmentation to complete foreground segmentation. Two algorithms are explored, Graph Cut and Relaxation Labeling. The disadvantage of foreground IR illumination can be compensated by background illumination. Keywords Video segmentation, Graph Cut, Relaxation Labeling, Infrared Illumination 1. Introduction Many tasks in computer vision involve foreground-background video segmentation. One of the major applications is video conferencing, where there is a need to replace the background to create the impression of a virtual meeting. While there are some excellent algorithms solving this problem, based either on stereo [1] or motion [2], they all have problems with illumination changes, with large moving objects in the background and with high computational costs. In this paper, we seek a solution to bi- layer segmentation problem by fusing infrared, color and contrast information. The challenges solved by our proposed system include real-time processing, automatic segmentation, and robustness to illumination changes and dynamic backgrounds. We investigate two ways of employing IR information by illuminating different parts of the scene: foreground IR illumination and background IR illumination, where foreground and background are illuminated by the IR illuminator. In the foreground IR illumination, the resulting video sequences are fed into the segmentation algorithm to complete foreground segmentation. We explored two algorithms, Graph Cut and Relaxation Labeling. A technical demo is produced using the Graph Cut algorithm. 2. Foreground IR Illumination 2.1 Data Acquisition In an earlier paper [3], we have described a data acquisition unit for producing automatically synchronized IR video and color video. The same data acquisition unit is used in this paper (see [3] for more details). By lighting the foreground with the IR illuminator, the resulting IR image is bright in the region closer to IR illuminator and dark in the background. The IR image can be easily registered with the video image so that there is a pixel-by- pixel correspondence between the IR image and the color image. A binary foreground template can be obtained by thresholding the IR image, as shown in Figure 1. (a) (b) (c) Figure 1. (a) Original produced IR image and color image pair (b) Registered IR image (c) Foreground template 2.2 Graph Cut (a) (b) (c) Figure 2. (a) Pentamap (b) Graph construction with T-links (c) Graph construction with N-links We introduced the concept of a pentamap in [3]. The pentamap partitions an image into five regions as shown in Figure 2(a). The red and blue areas result directly from the foreground template. We grow this template by a strip of width w, and predict that missing foreground parts are in this strip region, called “unknown region”. The value of w can be determined from the IR configuration. Any area beyond the unknown region is predicted to belong to the background area, which is marked by pink and green regions. The blue area is a thin boundary strip extracted from the foreground template, and is used to build the foreground color model, which is a Gaussian Mixture Model (GMM) in our