Signal Processing: Image Communication 18 (2003) 185–202 Rigid and non-rigid 3D motion estimation from multiview image sequences $ N. Ploskas a , D. Simitopoulos a , D. Tzovaras b , G.A. Triantafyllidis a, *, M.G. Strintzis a,b,1 a Information Processing Laboratory, Electrical and Computer Engineering Department, Aristotle University of Thessaloniki, 54124 Greece b Informatics and Telematics Institute, 1st Km Thermi-Panorama Road, Thermi-Thessaloniki 57001, Greece Received 23 November 2001; received in revised form 1 April 2002; accepted 30 October 2002 Abstract Multiview image sequence processing has been the focus of considerable attention in recent literature. This paper presents an efficient technique for object-based rigid and non-rigid 3D motion estimation, applicable to problems occurring in multiview image sequence coding applications. More specifically, a neural network is formed for the estimationoftherigid3Dmotionofeachobjectinthescene,usinginitiallyestimated2Dmotionvectorscorresponding to each camera view. Non-linear error minimization techniques are adopted for neural network weight update. Furthermore, a novel technique is also proposed for the estimation of the local non-rigid deformations, based on the multiview camera geometry. Experimental results using both stereoscopic and trinocular camera setups illustrate and evaluate the proposed scheme. r 2002 Elsevier Science B.V. All rights reserved. Keywords: Motion estimation; Rigid/non-rigid; Multiview 1. Introduction Depthunderstandingisanimportantelementof enhanced perception and tele-presence in image communication [10,29,3].Adirectwayofinferring the depth information is provided by stereo and multi-ocular vision [4,22]. Stereoscopic, or in general multiview video, can provide more vivid and accurate information about the scene struc- ture than simple video. Therefore, multiview video processing has been the focus of considerable attention in recent literature [8,14,21,24,25].Ina multiview image sequence, each different view is recorded with a difference in the observation angle, creating an enhanced 3D feeling to the observer, and increased ‘‘tele-presence’’ e.g. in teleconferencing. Model-based coding has long attracted consid- erable attention as a promising alternative to block-based encoding for the analysis and coding $ This work was supported by the EU project IST ‘‘HI- SCORE’’. *Corresponding author. Informatics and Telematics Insti- tute, 1st Km Thermi-Panorama Road, Thermi-Thessaloniki, 57001 Greece. E-mail addresses: ploskas@dion.ee.auth.gr (N. Ploskas), dsim@olympus.ee.auth.gr (D. Simitopoulos), dimitrios.tzovar- as@iti.gr (D. Tzovaras), gatrian@iti.gr (G.A. Triantafyllidis), strintzi@eng.auth.gr (M.G. Strintzis). 1 Also for correspondence. 0923-5965/03/$-see front matter r 2002 Elsevier Science B.V. All rights reserved. doi:10.1016/S0923-5965(02)00131-5