1051-8215 (c) 2016 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission. See http://www.ieee.org/publications_standards/publications/rights/index.html for more information. This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI 10.1109/TCSVT.2017.2706197, IEEE Transactions on Circuits and Systems for Video Technology JOURNAL OF L A T E X CLASS FILES, VOL. 14, NO. 8, AUGUST 2015 1 Copy and Paste: Temporally Consistent Stereoscopic Video Blending Zongji Wang, Xiaowu Chen, Senior Member, IEEE, and Dongqing Zou Abstract—We propose a novel method of stereoscopic video blending, targeting at achieving temporal disparity and color consistency. Video blending is one of the most frequent and important tasks in video editing, which is true in stereoscopic video editing too. However, it is more difficult to achieve temporally consistent blending for stereoscopic videos compared to blending of monocular videos since there is a more channel, namely disparity, to be considered except color channels in stereo- scopic videos. Toward this end, two algorithms are proposed in this paper for temporally consistent videos blending. One is a temporally-coherent mask propagation mechanism for selecting a source video patch clip from the source stereoscopic video; the other is a temporal blending algorithm, which seeks to adjust the shape of the source video patch clip so as to keep consistent with disparities of the target stereoscopic video. We show various results on numerous examples to demonstrate the effectiveness and efficiency of our method. Index Terms—temporally consistent, stereoscopic videos, dis- parity map, blending. I. I NTRODUCTION With the rapid development of stereoscopic 3D movies and increasing availability of internet stereoscopic data, the de- mand for stereo image and video editing techniques increases. Stereo image and video editing is an emerging research area and it is evolved from ordinary image and video processing. However, although the traditional image and video editing methods have been well-studied in image processing, it is not proper to directly apply existing 2D image editing methods to the stereoscopic media since there is an additional dimension of disparity which should be considered when doing editing. Video blending is one of the most frequent and important editing tasks in video editing, which is true in stereoscopic video editing too. The core of video blending is to ensure the temporal consistency during color blending. However, except the requirement for color consistency, stereoscopic video blending has its own challenges. On the one hand, the disparity of the inserted video patch clip must be adjusted so as to match the disparity of the target stereoscopic video while guaranteeing the temporal consistency of blended disparity. On the other hand, the projected shape of the blended video patch clip should be well warped according to the generated This work was supported in part by the National Natural Science Foundation of China under Grant 61325011, Grant 61532003. (Corresponding author: Xiaowu Chen.) The authors are with the State Key Laboratory of Virtual Reality Tech- nology and Systems, School of Computer Science and Engineering, Bei- hang University, Beijing 100191, China (e-mail: wzjgintoki@buaa.edu.cn; chen@buaa.edu.cn; zoudq@buaa.edu.cn) disparity maps to avoid distortion. Although some existing image editing methods [1], [2], [3], [4], [5] obtain impressive results on 2D or stereoscopic image editing, the temporal consistency of video is not considered. Thus these methods can not be directly applied to stereoscopic video. To address these challenges, we propose a novel method for temporally consistent stereoscopic video blending, as Figure 1 shows. Different from previous methods, we first extract a temporally consistent stereoscopic video clip to be blended. Then, the disparity and color of the extracted stereoscopic video patch are adjusted in the form of multi-frame and multi-view simultaneously, to blend the extracted patch into the target video, yielding a temporally consistent blended stereoscopic video. The disparity of the inserted stereoscopic video region clip is important to ensure a temporally consistent composition. From previous work [1] we can see that the stereoscopic image or video blending is sensitive to the color and disparities of pixels lying on the contour of input region. Warping the shape of the input region may result in significant differences in color and disparity, leading to a temporally inconsistent composition effect. Thus, temporally-coherent stereoscopic video blending requires a color and disparity temporally- coherent inserted stereoscopic video clip. Generally speaking, neighboring frames have similar feature points, which helps to estimate the temporal relationship between frames. Inspired by that, we extract a stereoscopic video clip through feature matching among consecutive (previous and next) and adjacent (left and right) frames. Specifically, we first construct a trian- gular mesh for the first frame, and then apply feature matching, to propagate the mesh to the next frame while considering the relationship between the paired left and right view frames in the source stereoscopic video. Editing the stereoscopic video view by view will result in temporally inconsistent results. The relationship between views, which encodes the correspondence of pixels, acts as a key factor to stereoscopic video editing. Blending two stereo- scopic videos with multi-frame and multi-view simultaneously, which intrinsically encourages pixels with similar color and disparity to have similar blending results, enables us to lever- age the relationships between views and frames, to obtain a temporally-coherent composition result compared to blending frame by frame and view by view. Moreover, our color and disparity temporally-coherent video clip extraction algorithm can feed a consistency prior of color and disparity into the disparity blending algorithm, which further enforces the tem- Copyright c  2017 IEEE. Personal use of this material is permitted. However, permission to use this material for any other purposes must be obtained from the IEEE by sending an email to pubs-permissions@ieee.org.