EUROGRAPHICS 2011/ N. John and B. Wyvill STAR – State of The Art Report A Survey on Video-based Graphics and Video Visualization R. Borgo 1† , M. Chen 1 , B. Daubney 1 , E. Grundy 1 , H. Jänicke 3 , G. Heidemann 2 , B. Höferlin 2 , M. Höferlin 2 , D. Weiskopf 2 , X. Xie 1 1 Swansea University, UK 2 Stuttgart University, Germany 3 Heidelberg University, Germany Abstract In recent years, a collection of new techniques, which deal with videos as the input data, emerged in computer graphics and visualization. In this survey, we report the state of the art in video-based graphics and video visualiza- tion. We provide a comprehensive review of techniques for making photo-realistic or artistic computer-generated imagery from videos, as well as methods for creating summary and/or abstract visual representations to reveal important features and events in videos. We propose a new taxonomy to categorize the concepts and techniques in this newly-emerged body of knowledge. To support this review, we also give a concise overview of the major advances in automated video analysis, as some techniques in this field (e.g., feature extraction, detection, tracking and so on) have been featured in video-based modeling and rendering pipelines for graphics and visualization. 1. Introduction Until recently, videos have largely been used only as an out- put medium in computer graphics and visualization. The rapid advance of digital technologies has resulted in an ex- plosion of video data, which stimulates the needs for cre- ating computer graphics and visualization from videos. In this survey, we report the emergence of a new collection of graphics and visualization techniques, which deal with videos as the input data. Video-based graphics is concerned with the manipulation and rendering graphical models, which are built from video data, instead of, or in addition to, traditional object repre- sentations. Its primary aim is to make creative computer- generated imagery from videos for artistic appreciation and entertainment. There are two main strands in this field, video re-fashioning and video-based scene modeling. The former typically involves manipulation of the geometrical entities (e.g., object shape and distribution) and optical attributes (e.g., lighting, color) in an input video, and produces a new video that captures the essence of the input video but in an expressive art form, such as relighting the video scene with imaginary lights or mimicking hand-drawn cartoon anima- tion. The latter typically involves reconstruction a 3D object † Authors are listed in alphabetical order. Contact author is Dr. Rita Borgo, email: r.borgo@swansea.ac.uk or scene model captured by the input video, allowing such a model to be manipulated, combined with other models, and rendered in the same way as conventional graphical models. The main thrust for video-based graphics has been consumer multimedia applications, and the films and games industries. Video visualization is concerned with the creation of a new visual representation from an input video to reveal im- portant features and events in the video. It typically extracts meaningful information from a video and conveys the ex- tracted information to users in abstract or summary visual representations. Video visualization is not intended to pro- vide fully automatic solutions to the problem of making de- cision about the contents of a video. Instead, it aims at of- fering a tool that assists users in their intelligent reasoning while removing the burden of viewing videos. This aim justi- fies deflection from creation of realistic imagery, and allows simplifications and embellishments, if they improve the de- sired understanding of the input video. In many ways, the subject of video visualization encompasses some aspects of video-based graphics. Nevertheless, the development of the subject has been heavily influenced by many applications in science, medicine, sports and security. There is a huge collection of literature in the fields of image processing, computer vision, and multimedia tech- nology. Automated video analysis encompasses a variety of techniques, ranging from low-level processing techniques c  The Eurographics Association 2011.