A Probabilistic Framework for Unsupervised Evaluation and Ranking of Image Segmentations Mustafa Jaber (1) , Sreenath Rao Vantaram (1) , and Eli Saber (1,2) 1 Chester F. Carlson Center for Imaging Science, Rochester Institute of Technology, Rochester, NY 14623 USA 2 Department of Electrical and Microelectronic Engineering, Rochester Institute of Technology, Rochester, NY 14623 USA {mij7272, sxv9436, esseee}@rit.edu Abstract- In this paper, a Bayesian Network (BN) framework for unsupervised evaluation of image segmentation quality is proposed. This image understanding algorithm utilizes a set of given Segmentation Maps (SMs) ranging from under- segmented to over-segmented results for a target image, to identify the semantically meaningful ones and rank the SMs according to their applicability in image processing and computer vision systems. Images acquired from the Berkeley segmentation dataset along with their corresponding SMs are used to train and test the proposed algorithm. Low-level local and global image features are employed to define an optimal BN structure and to estimate the inference between its nodes. Furthermore, given several SMs of a test image, the optimal BN is utilized to estimate the probability that a given map is the most favorable segmentation for that image. The algorithm is evaluated on a separate set of images (none of which are included in the training set) wherein the ranked SMs (according to their probabilities of being acceptable segmentation as estimated by the proposed algorithm) are compared to the ground-truth maps generated by human observers. The Normalized Probabilistic Rand (NPR) index is used as an objective metric to quantify our algorithm’s performance. The proposed algorithm is designed to serve as a pre-processing module in various bottom-up image processing frameworks such as content-based image retrieval and region-of-interest detection. Keywords: Segmentation Evaluation, Bayesian Networks, Image Understanding. I. INTRODUCTION Many image segmentation algorithms have been proposed in the recent years to group image pixels yielding semantically meaningful maps. Different mathematical and physical theories have been employed to develop these techniques as summarized in the surveys [1] and [2]. The majority of the state-of-the-art image segmentation techniques yield in a single map for any target image with the assumption that it is the optimal segmentation under certain conditions. Spatial and spectral features are used as optimization criteria to generate the best (optimal) segmentation map. This may include empirically setting some thresholds or setting a number for expected segments in the output map. Furthermore, the segmentation algorithms in the literature are usually evaluated and benchmarked against other algorithms on a limited set of test images. This comparison may yield that a certain methodology of segmentation performs better on a set of images but this does not mean that this “winner” algorithm will definitely generate the optimum Segmentation Map (SM) for any target image. In this paper, we try to overcome these limitations by developing a framework to identify a number of semantically acceptable segmentation maps of a test image, and estimate the probability of each map of being the optimal segmentation of the target image. The probability values are used to rank the segmentations in accordance to their usability. Methodologies for evaluating the performance of image segmentation algorithms are having more attention in the recent years. They are essential to systematically determine the merit, worth, and significance of the segmentation techniques. Objective and subjective segmentation evaluation methods are found in the literature and are summarized in the survey by Zhang et al. [3]. More resent segmentation evaluation techniques are found in [4], [5]. The method in [4] is based on genetic algorithms which were used to minimize feature criteria over different segmentation maps. The search space is defined as the label of each pixel of the test image. Furthermore, the technique proposed in [5] utilized energy formulation to generate multi-scale criteria. However, the level of detail expected by the evaluation criterion is left to the user to select. In this paper, an image understanding algorithm for ranking segmentation maps of an arbitrary image into different levels according to their usefulness is proposed. An “acceptable” segmentation is a map that depicts a semantically meaningful partitioning of image regions. The proposed algorithm has two phases: training phase which utilizes ground truth segmentation maps to build a Bayesian Network (BN) and a testing phase. The proposed algorithm includes modules for image segmentation, feature extraction, discretization, and probabilistic reasoning. The proposed algorithm generates multiple segmentation maps utilizing the algorithm in [6] and ranks them according to their estimated probability of being acceptable in image processing and computer vision systems. The remainder of this paper is organized as follows. The process of generating ground-truth SMs is discussed in Section II. The proposed algorithm is introduced in Section III. Results and performance evaluation of the proposed algorithm are shown in Section IV. Finally, conclusions and future work are provided in Section V.