A real-time data fusion system updating 3D organ shapes using color information from multi-directional cameras Miyuki Uematsu a, * , Naoki Suzuki b , Asaki Hattori b , Yoshito Otake b , Shigeyuki Suzuki b , Mitsuhiro Hayashibe b , Susumu Kobayashi c , Akihiko Uchiyama a a Major in Integrative Bioscience and Biomedical Engineering, Graduate School of Science and Engineering, Waseda University, 3-4-1 Okubo, Shinjuku, Tokyo 169-8555, Japan b Institute for High Dimensional Medical Imaging, Jikei University School of Medicine, Tokyo, Japan c Department of Surgery, Jikei University School of Medicine, Tokyo, Japan Abstract. We have been developing a data fusion system, which allows surgeons to visualize the inner structures of organs during liver surgery. The liver alters its shape dynamically during an operation. In this system, we used stereo cameras to track intraoperative liver deformation. Color markers were attached on to the liver surface and tracked by the stereo cameras. Multi-directional camera views compensated for the lack of surface data during surgical procedures, even if the surface data could not be acquired from one directional camera view. A three-dimensional (3D) organ model that represented inner structures was reconstructed from preoperative image data sets and deformed by the intraoperative data. The model was updated on the video images of the surgical field. We examined the feasibility of this data fusion system in a liver phantom examination and in an in-vivo experiment with a pig liver. D 2004 Published by Elsevier B.V. Keywords: Data fusion; Open surgery; Stereovision 1. Introduction Image-guided surgical systems (IGS) have been developed in the field of neurosur- gery. Clinical trials have reported that IGS is a major benefit for surgeons when it comes to visualizing the spatial relationships between vasculatures and tumors during the operation. Recently, some researchers have attempted to utilize this technology for liver surgery. It was considered problematic due to the liver’s propensity to alter shape during surgery. However, this can be dealt with by recent advances in deformable 3D organ models in the field of the liver surgical simulators [1–3]. 0531-5131/ D 2004 Published by Elsevier B.V. doi:10.1016/j.ics.2004.03.328 * Corresponding author. Tel.:+81-3-3480-1151; fax: +81-3-5438-8380. E-mail address: miyuki@fuji.wesada.jp (M. Uematsu). www.ics-elsevier.com International Congress Series 1268 (2004) 741 – 746