924 IEEE TRANSACTIONS ON IMAGE PROCESSING, VOL. 12, NO. 8, AUGUST 2003 Relevance Feedback in Content-Based Image Retrieval: Bayesian Framework, Feature Subspaces, and Progressive Learning Zhong Su, Hongjiang Zhang, Stan Li, and Shaoping Ma Abstract—Research has been devoted in the past few years to rel- evance feedback as an effective solution to improve performance of content-based image retrieval (CBIR). In this paper, we propose a new feedback approach with progressive learning capability com- bined with a novel method for the feature subspace extraction. The proposed approach is based on a Bayesian classifier and treats posi- tive and negative feedback examples with different strategies. Pos- itive examples are used to estimate a Gaussian distribution that represents the desired images for a given query; while the nega- tive examples are used to modify the ranking of the retrieved can- didates. In addition, feature subspace is extracted and updated during the feedback process using a Principal Component Analysis (PCA) technique and based on user’s feedback. That is, in addition to reducing the dimensionality of feature spaces, a proper subspace for each type of features is obtained in the feedback process to fur- ther improve the retrieval accuracy. Experiments demonstrate that the proposed method increases the retrieval speed, reduces the re- quired memory and improves the retrieval accuracy significantly. Index Terms—Bayesian estimation, content-based image re- trieval, principal component analysis (PCA), relevance feedback (RF). I. INTRODUCTION C ONTENT-BASED image retrieval (CBIR) is a process to find images similar in visual content to a given query from an image database. It is usually performed based on a com- parison of low level features, such as color, texture or shape features, extracted from the images themselves. While there is much research effort addressing content-based image retrieval issues [1], [11], [19], the performance of content-based image retrieval methods are still limited, especially in the two aspects of retrieval accuracy and response time. The limited retrieval accuracy is because of the big gap be- tween semantic concepts and low-level image features, which is the biggest problem in content-based image retrieval. For ex- ample, for different queries, different types of features have dif- ferent significance; an issue is how to derive a weighting scheme Manuscript received July 2, 2001; revised March 26, 2003. The associate ed- itor coordinating the review of this manuscript and approving it for publication was Dr. Christine Guillemot. Z. Su is with the State Key Lab of Intelligent Tech. and Systems, Tsinghua University, Beijing 100084, China (e-mail: suzhong_bj@hotmail.com). H. Zhang is with the Microsoft Research Asia, Beijing 100080, China (e-mail: hjzhang@microsoft.com). S. Li is with the Microsoft Research Asia, 5F, Beijing Sigma Center, Beijing 100080, China (e-mail: szli@microsoft.com). S. Ma is with the State Key Lab of Intelligent Tech. and Systems, Tsinghua University, Beijing 100084, China (e-mail: msp@tsinghua.edu.cn). Digital Object Identifier 10.1109/TIP.2003.815254 to balance the relative importance of different feature type and there is no universal formula for all queries. The relevance feed- back technique can be used to bridge the gap [3], [12], [13], [17], [24]–[26]. Relevance feedback, originally developed for information retrieval [16], is a supervised learning technique used to improve the effectiveness of information retrieval systems. The main idea of relevance feedback is using positive and negative examples provided by the user to improve the system’s performance. For a given query, the system first retrieves a list of ranked images according to predefined similarity metrics, which are often defined as the distance between feature vectors of images. Then, the user selects a set of positive and/or negative examples from the retrieved images, and the system subsequently refines the query and retrieves a new list of images. The key issue is how to incorporate positive and negative examples to refine the query and how to adjust the similarity measure according to the feedback. The original relevance feedback method, in which the vector model [1], [20], [21] is used for document retrieval, can be il- lustrated by the Rocchio’s formula [16] as (1) where , and are suitable constants and and are the number of documents in and , respectively. That is, for a given initial query , and a set of relevant documents and nonrelevant documents given by the user, the optimal new query, , is the one that is moved toward positive example points and away from negative example points. This technique is also implemented in many content-based image retrieval sys- tems [9], [12]. Experiments show that the retrieval performance can be improved considerably by using this approach. Gen- erally speaking, previous relevance feedback methods can be classified into two approaches: “the weighing approach” and “the probability approach.” Most existing work in content-based image retrieval uses the former approach. The weighting method [9], [17], [19] associates larger weights with more important dimensions and smaller weights with less important ones. For example, [19] generalizes a relevance feedback framework of the low-level feature-based relevance feedback methods. An ideal query vector for each 1057-7149/03$17.00 © 2003 IEEE