Signal Processing: Image Communication 20 (2005) 413–434 Effective adaptation of multimedia documents with modality conversion Truong Cong Thang, Yong Ju Jung, Yong Man Ro à Multimedia Group, Information and Communications University (ICU), Yuseong, Daejon, PO Box. 77, 305-732, Republic of Korea Received 18 October 2004; accepted 8 March 2005 Abstract Besides content scaling, modality conversion is an important aspect of content adaptation. In this paper, we study modality conversion of a multimedia document under the constraints of available resource and human factor (user preference). We first formulate the content adaptation process of a multimedia document as a general constrained optimization problem and then extend it to effectively support modality conversion. To represent conversion boundaries between different modalities, we propose the overlapped content value (OCV) model that relates the content value of different modalities with resources. Also, the human factor is specified in the form of modality conversion preference, and then integrated into the framework by modifying the OCV model. We apply the Viterbi algorithm of dynamic programming and its fast approximation to the optimization problem with a practical resource constraint, namely the total datasize. Experiments demonstrate that modality conversion (in combination with content scaling) brings a wider range of adaptation for QoS support. Moreover, the proposed approach is shown to be effective to apply in practice. r 2005 Elsevier B.V. All rights reserved. Keywords: Content adaptation; MPEG-21 DIA; Modality conversion; User preference; Multimedia processing 1. Introduction Content adaptation is a key solution to support the quality of service (QoS) for multimedia services over heterogeneous environments [9,7]. More specifically, multimedia documents are adapted to meet various constraints from terminals, net- work connections, and user preference, while providing the best possible presentation to the user. Content adaptation has two major aspects: one is modality conversion (also called transmod- ing), which converts contents from one modality to a different modality (e.g. from video to image or text); the other is content scaling, which changes the amount of resource (e.g. datasize) and so the quality of the contents without converting their modalities. The term resource here means a kind of computing support needed for delivering contents ARTICLE IN PRESS www.elsevier.com/locate/image 0923-5965/$-see front matter r 2005 Elsevier B.V. All rights reserved. doi:10.1016/j.image.2005.03.004 à Corresponding author. Tel.: +82428666279; fax: +82428666245. E-mail address: yro@icu.ac.kr (Y.M. Ro).