Superresolution reconstruction of a video captured by a vibrated time delay and integration camera Oren Haik Yitzhak Yitzhaky Ben-Gurion University Department of Electro-Optics Engineering Beer-Sheva 84105 POB 653 Israel E-mail: itzik@ee.bgu.ac.il Abstract. Various applications such as industrial product inspec- tion or low signal-to-noise situations (as in thermal imaging) employ a time delay and integration (TDI) scanning imaging technique. Due to common vibration sources such as the camera platform motion or the thermal detector’s cooling system, the acquired image may be degraded by severe shift-variant geometric distortions and motion blur. We use these vibrations in terms of superresolution to create an improved high-resolution video sequence from the degraded lower resolution sequence, in two main stages: subpixel motion es- timation with respect to translations and rotations, used for point spread function (PSF) estimation, followed by an efficient implemen- tation of the projection onto convex sets (POCS) method. We gen- eralize and considerably improve a previous technique for restora- tion of a single image captured by a translational vibrated staggered-TDI camera (Hochman et al., 2004). The proposed method is implemented with both simulated videos and real de- graded thermal videos. A comparative analysis shows an advantage of the proposed method over others in restoring the vibrated videos. © 2006 SPIE and IS&T. DOI: 10.1117/1.2194042 1 Introduction A line-scan imaging system produces a 2-D image, line by line sequentially, via a single vector of contiguous sensing elements that scans the scene of interest. Characteristics of the scanning vector, such as the number of elements which can range up to thousands, and the integration exposure time can be altered to suit the requirements of the application. 1 Although such a mechanism can feature very high spatial resolution, it may result in a low signal-to- noise ratio SNR in common situations such as low avail- able scene illumination or thermal imaging. Since longer integration time increases both signal and blur, time delay and integration TDI mode 2,3 is used, in which the scan- ning element has several adjacent vectors TDI stages, where multiple short exposures of the object are integrated throughout the TDI stages. This results in an SNR improve- ment of N 1/2 , where N is the number of TDI stages. 3 Due to physical constraints 4 the TDI sensor elements may have a staggered structure, as shown in Fig. 1. The staggered TDI imager is basically composed of a pair of spatially shifted imaging arrays, one captures the odd im- age rows also called the odd field, while the other cap- tures the even rows the even field, forming an interleaved image with full spatial resolution perpendicular to the scan direction. Typically its height number of rows, M is larger than its length number of columns or TDI stages, N, and a 2-D image is obtained by mechanically scanning the image plane. Imaging systems are frequently affected by vibrations caused by various sources such as servoengines, platform vibrations airplanes, tanks, etc., and electrical cooling systems for thermal imaging systems. Although most of these vibrations can be reduced by a proper design such as a mechanical tension against a roller, a residual motion due to vibrations always exists. These vibrations blur the image and cause image distortions such as comb effects and geo- metric warps. Most of the researches that dealt with resto- ration of images degraded by vibrations considered mainly staring nonscanning cameras. 5–8 This paper addresses restoration of a video of a station- ary scene, captured by a vibrated staggered TDI camera. As a result of the vibrations and the scanning mechanism just described, the resulting distortion is not shift-invariant but rather, it is spatially random according to the vibration spectrum, 9 hence, common shift-invariant image filtering restoration techniques are inappropriate. Nevertheless, the information loss resulting from vibrations and from the fi- Paper 05130R received Jul. 7, 2005; revised manuscript received Nov. 17, 2005; accepted for publication Dec. 19, 2005; published online May 17, 2006. This paper is a revision of a paper presented at the SPIE conference on Applications of Digital Image Processing XXVII, Aug. 2004, Denver, Colorado. The paper presented there appears unrefereed in SPIE Pro- ceedings Vol. 5558. 1017-9909/2006/152/023006/12/$22.00 © 2006 SPIE and IS&T. Fig. 1 Imaging process of a staggered-TDI camera. Journal of Electronic Imaging 15(2), 023006 (Apr–Jun 2006) Journal of Electronic Imaging Apr–Jun 2006/Vol. 15(2) 023006-1