1004 IEEE TRANSACTIONS ON MULTIMEDIA, VOL. 9, NO. 5, AUGUST 2007 Event Dynamics Based Temporal Registration Meghna Singh, Anup Basu, Senior Member, IEEE, and Mrinal Mandal, Senior Member, IEEE Abstract—Temporal registration is the establishment of cor- respondence between two (or more) temporal frames of video sequences, or 3-D volume data. In this paper, we propose to use event dynamics, a property that is inherent to an event and is thus common to all acquisitions of the event, for both global and local temporal registration of video sequences in order to generate high temporal resolution video. We compare our approach to a widely used linear interpolation based temporal registration algorithm and demonstrate that in the case of low temporal acquisition rate, a global event dynamics based approach, such as ours, has smaller temporal registration error. We also present a unique application of our work in solving 3-D high temporal resolution medical data visualization problem. Index Terms—Event dynamics, temporal registration and med- ical data visualization. I. INTRODUCTION A LL DIGITAL DATA acquisition techniques essentially acquire discrete samples of a fundamentally continuous world. Often, limited by current technology, we can only collect and process a limited amount of information from the events that occur around us. For example, generic video cameras can capture information at frame rates of 30 frames per second (fps); a fast magnetic resonance imaging (MRI) protocol can capture only 6–7 fps of soft tissue motion in the human body. In order to recreate the true event that occurred, most researchers use multiple acquisition sources and offset their capture so that correlated but discrete samples are acquired. In this paper, we address the unique problem: “What if you cannot have more than a single acquisition source?” To solve this issue, one approach is to reconstruct information by interpolating between low-resolution discrete samples available in a single set of data. Interpolation is a poor solution when few samples are present, as the interpolated data may be completely inaccurate when compared to the real event. The unique approach we propose is to use repetitions of the same event (which may or may not be replicable each time) and use event dynamics to temporally register and reconstruct the samples in a multidimensional space. This temporal registration will generate a high-resolu- tion data set, which can be used further for applications such as Manuscript received September 1, 2006; revised February 7, 2007. This work was supported in part by the Natural Sciences and Engineering Research Council of Canada (NSERC) under Grant G121210634 and by the Alberta Science and Research Authority (ASRA). The associate editor coordinating the review of this manuscript and approving it for publication was Dr. Yoshihisa Shinagawa. M. Singh and M. Mandal are with the Department of Electrical and Computer Engineering, University of Alberta, Edmonton, AB, Canada, T6G 2V4 (e-mail: meghna@ece.ualberta.ca; mandal@ece.ualberta.ca). A. Basu is with the Department of Computing Science, University of Alberta, Edmonton, AB, Canada, T6G 2E8 (e-mail: anup@cs.ualberta.ca). Color versions of one or more of the figures in this paper are available online at http://ieeexplore.ieee.org. Digital Object Identifier 10.1109/TMM.2007.898937 robust object tracking, super-high resolution video generation or medical image visualization. Temporal registration attempts to establish correspondence between two (or more) time-varying datasets such as video sequences, or 3-D volume data. In the past, temporal regis- tration has been achieved by using either specific timestamp information from an external source or information derived directly from the video data by identifying key points. In both these cases, global and local misalignments are computed by minimizing a matching criterion. While global alignment allows the video sequences to be roughly registered, correc- tion of local misalignment is essential as it allows subframe temporal registration. Current techniques for correcting local misalignment are based on linear interpolation or on splines. Linear interpolation leads to exact fitting which increases the dependence of the fit on the accuracy of the trajectory points. If feature extraction or coordinate selection is not robust, error is introduced into the registration. Splines on the other hand lead to a large solution space and nonunique registration. We propose to use event dynamics, a property that is inherent to an event and is thus common to all acquisitions of the event, to integrate multiple low-temporal resolution acquisitions to- gether to generate high-temporal resolution data. First let us de- fine an ‘event’. An event can be defined as the occurrence of something important at a certain spatial location, over an in- terval of time. Thus, the two parameters important in defining an event are its spatial location and its temporal range. It is our hypothesis that any event in multidimensional space will gen- erate distinct spatiotemporal patterns, and while these patterns may not be identical over multiple repetitions, they will have a high degree of correlation. Space-time patterns of image ele- ments (such as single pixels or regions of an image) can be rep- resented as motion models of the elements. This will allow the parameterization of events occurring in spatio-temporal space. Low-resolution capture limits us to sample only a few represen- tative space–time points of these motion patterns. However, if sufficient spatiotemporal patterns of repeated instances of the same event are available, then the low-resolution samples can be used to generate a high-resolution space–time pattern of the event. This rest of this paper is organized as follows. In Section II and Section III, we discuss some contemporary works related to motion modeling and temporal registration respectively. In Section IV we discuss our algorithm for matching event dynamics based curves. We also present a comparative anal- ysis of a contemporary algorithm proposed by Caspi et al. [5], [10] and submit a case study where their algorithm gives erroneous results. In Section V we present our experimental setup and results with real and synthetic data; we also submit results obtained with the Caspi algorithm on the same data. In Section VI we present a novel application of our work in registering and generating high resolution MRI sequences for 1520-9210/$25.00 © 2007 IEEE