MUKHERJEE et al.: TERMINAL AND NETWORK QUALITY OF SERVICE 1 Abstract—In order to cater to diversity of terminals and networks, efficient and flexible adaptation of multimedia content in the delivery path to end consumers is required. To this end, it is necessary to associate the content with metadata that provides the relationship between feasible adaptation choices and various media characteristics obtained as a function of these choices. Further, adaptation is driven by specification of terminal, network, user preference or rights based constraints on media characteristics that are to be satisfied by the adaptation process. Using the metadata and the constraint specification, an adaptation engine can take an appropriate decision for adaptation, efficiently and flexibly. MPEG-21 Part 7 entitled Digital Item Adaptation standardizes among other things the metadata and constraint specifications that act as interfaces to the decision-taking component of an adaptation engine. This paper presents the concepts behind these tools in the standard, show universal methods based on pattern search to process the information in the tools to make decisions, and present some adaptation use cases where these tools can be used. Index Terms—MPEG-21, Digital Item Adaptation, terminal and network constraints, decision-taking, adaptation, transcoding, requantization, rate shaping, scalable bit-streams. I. INTRODUCTION ETEROGENEOUS multimedia content delivery infrastructures and consumption devices present a huge obstacle in universal media access. Indeed, consumers use a growing variety of terminals to access multimedia content, over an equally diverse variety of networks with dynamically varying throughputs. To maximize consumer experience and ensure Quality of Service (QoS) commensurate with terminal and network capabilities and conditions, as well as user preferences, it is essential to adapt multimedia content in the delivery path to end consumers. Note here QoS is used loosely and does not correspond to network level guarantees. Additionally, the set of rich media content and formats to be delivered is growing fast. This justifies a drive towards adaptation engines or modules thereof that use a universal processing model – which do not need frequent upgrades to Manuscript received April 30, 2004, revised June 30, 2004. Part of this work was funded by the IST-OZONE project (IST-2000-30026). Debargha Mukherjee is with Hewlett Packard Laboratories, Palo Alto, CA, USA (phone: 650-236-8058; fax: 650-857-2951; e-mail: debargha@ hpl.hp.com). Eric Delfosse is with Interuniversity MicroElectronics Center (IMEC), Kapeldreef 75, B-3001 Leuven, Belgium (Phone: +32 16 28 10 68, Fax: +32 16 28 15 15, E-mail: delfosse@imec.be). Jae-gon Kim is with ETRI, Korea (e-mail: jgkim@etri.re.kr). Yong Wang is with the Electrical Engineering Department, University of Columbia University, NY, USA (e-mail: ywang@ee.columbia.edu). support new formats and can even support proprietary ones. Adaptation of various standardized formats has been extensively studied in recent years [1]-[9]. Invariably the focus of such work is adaptation efficiency, since full decoding followed by re-encoding with parameters so that the terminal and network constraints are met, is often infeasible from complexity and delay considerations. This includes rate adaptation and resolution conversion for Discrete Cosine Transform (DCT) coded images [1][2], rate adaptation and spatial and temporal resolution conversion for pre-encoded MPEG-1/2/4 videos [3][4][5], object based transcoding [6], and rate-distortion-complexity optimized transcoding [7]. In many of these cases, there is a compute intensive decision-taking involved for choosing the right set of parameters for adaptation that yields an adapted version of the content meeting terminal and network constraints. The adaptation efficiency can be greatly improved if this process could be simplified, in particular by providing some metadata that conveys pre-computed relationships between feasible adaptation parameters and media characteristics obtained by selecting them. This metadata is also the only means of providing information that cannot be directly obtained from a compressed bit-stream, such as distortion/fidelity measures with respect to the original uncompressed data. The decision- taking process then just uses the information in the metadata along with terminal and network constraints to make decisions, without requiring any information extraction through complex content manipulation. Furthermore, a universal processing model for the decision-taking process in an adaptation engine can be derived, so that descriptions and engines created by different parties can interoperate. Digital Item Adaptation (DIA) [10][11] is Part 7 of the interoperable Multimedia Framework currently being developed in the ISO/IEC MPEG standardization committee as MPEG-21 [12][13], and aims to standardize various descriptions, called tools, on the Terminal and Network key element, including the metadata supporting decision-taking and the constraint specifications as required for QoS. The rest of the paper is organized as follows. In Section II, the model for an adaptation engine is presented along with an introduction to various DIA components. Section III presents the decision-taking framework in detail. In Section IV, the optimization problem to be solved by a universal decision- taking process is described, along with some strategies for solving it. In Section V we show a variety of adaptation use cases involving various formats where the framework can be effectively employed to make adaptation decisions. Finally, conclusions and future directions are presented in Section VI. Terminal and Network Quality of Service Debargha Mukherjee, Member, IEEE, Eric Delfosse, Jae-Gon Kim, Member, IEEE, and Yong Wang, H