IEEE TRANSACTIONS ON BIOMEDICAL ENGINEERING, VOL. 60, NO. 12, DECEMBER 2013 3391 Assessing the Need for Referral in Automatic Diabetic Retinopathy Detection Ramon Pires ∗ , Member, IEEE, Herbert F. Jelinek, Member, IEEE, Jacques Wainer, Siome Goldenstein, Senior Member, IEEE, Eduardo Valle, and Anderson Rocha, Member, IEEE Abstract—Emerging technologies in health care aim at reducing unnecessary visits to medical specialists, minimizing overall cost of treatment and optimizing the number of patients seen by each doctor. This paper explores image recognition for the screening of diabetic retinopathy, a complication of diabetes that can lead to blindness if not discovered in its initial stages. Many previous reports on DR imaging focus on the segmentation of the retinal image, on quality assessment, and on the analysis of presence of DR-related lesions. Although this study has advanced the detection of individual DR lesions from retinal images, the simple presence of any lesion is not enough to decide on the need for referral of a patient. Deciding if a patient should be referred to a doctor is an es- sential requirement for the deployment of an automated screening tool for rural and remote communities. We introduce an algorithm to make that decision based on the fusion of results by metaclassifi- cation. The input of the metaclassifier is the output of several lesion detectors, creating a powerful high-level feature representation for the retinal images. We explore alternatives for the bag-of-visual- words (BoVW)-based lesion detectors, which critically depends on the choices of coding and pooling the low-level local descriptors. The final classification approach achieved an area under the curve of 93.4% using SOFT–MAX BoVW (soft-assignment coding/max pooling), without the need of normalizing the high-level feature vector of scores. Index Terms—Bag-of-visual-words (BoVW), diabetic retino- pathy, lesion detectors, metaclassification, referral, visual dictionaries. I. INTRODUCTION T HE development of computational systems that support specialists in diverse areas of health care has been the fo- cus of several studies [1]–[6]. The use of computational methods Manuscript received February 15, 2013; accepted August 4, 2013. Date of publication August 16, 2013; date of current version November 18, 2013. This work was supported in part by S˜ ao Paulo Research Foundation FAPESP under Grant 2010/05647-4 and Grant 2011/15349-3, National Counsel of Technolog- ical and Scientific Development (CNPq) under Grant 307018/2010-5 and Grant 304352/2012-8), and Microsoft. Asterisk indicates corresponding author. ∗ R. Pires is with the Institute of Computing, University of Campinas, Camp- inas 13083-852, Brazil (e-mail: pires.ramon@students.ic.unicamp.br). H. F. Jelinek is with the Department of Biomedical Engineering, Khalifa University, Abu Dhabi 127788, UAE, and also with the Australian School of Advanced Medicine, Macquarie University, North Ryde, N.S.W. 2113, Australia (e-mail: hjelinek@csu.edu.au). J. Wainer, S. Goldenstein, and A. Rocha are with the Institute of Computing, University of Campinas, Campinas 13083-852, Brazil (e-mail: siome@ic.unicamp.br; wainer@ic.unicamp.br; anderson@ic.unicamp.br). E. Valle is with the School of Electrical and Computing Engineering, Uni- versity of Campinas, Campinas 13083-852, Brazil (e-mail: dovalle@dca.fee. unicamp.br). Color versions of one or more of the figures in this paper are available online at http://ieeexplore.ieee.org. Digital Object Identifier 10.1109/TBME.2013.2278845 that aid in the diagnosis of disease has contributed significantly to improve the quality of life of patients. In this context, several computational systems have been proposed (see, e.g., [1]–[3] and [7]) for dealing with complications related to Diabetes Mellitus. According to the International Diabetes Federation, 1 diabetes will nearly double to 552 million people by 2030 [8]. Diabetes- related complications are also increasing in prevalence including diabetic retinopathy, which currently affects 2–4% of people with diabetes [9], [10] and is the main cause of blindness in the 20–74 age group in developed countries [11]. The development of a unified screening system that simulta- neously identifies several different DR-related lesions has been described using a bag-of-visual-words (BoVW) model based upon visual dictionaries [1], [2], [12]. This model needs a visual dictionary for each type of lesion, and hence, a specific classifier is required for each type of lesion. To decide on the level of DR progression (from mild to severe), or the need for referral, one must combine the separate classifiers into a unified model. In this paper, we propose a method that recommends referring a patient with diabetes for diabetic retinopathy assessment based on the image classification outcome, which is especially useful in remote and rural areas. The method captures retinal images from nonmydriatic or mydriatic cameras, evaluates the images in real time, and suggests whether or not the patient requires a review by an ophthalmic specialist within one year after the screening. The method consists of 1) detecting individual retinal anomalies and extracting the appropriate assessment scores, and 2) classifying the image as referable versus nonreferable by means of metaclassification techniques built upon the output of several lesion detectors. Different from [1], [2], and [12], we explore alternatives for the BoVW lesion detectors because the performance of BoVW depends critically on the choices of coding and pooling the low-level local descriptors and aim at characterizing the properties and signs related to each kind of lesion of interest. We organized the remainder of this paper in four sections. Section II describes the related work, while Section III explains our methodology for referral versus nonreferral classification. Section IV presents the experimental results for the proposed methods, and, finally, Section V concludes this paper. II. STATE OF THE ART The existence of a DR-related lesion in a retinal image does not necessarily indicate a vision-threatening sign that requires 1 http://www.idf.org/diabetesatlas/5e/diabetes 0018-9294 © 2013 IEEE