1 Apple detection in natural tree canopies from multimodal images J. P. Wachs 1,2 , H. I. Stern 2 , T. Burks 3 and V. Alchanatis 1 1 Institute of Agricultural Engineering, Agricultural Research Organization, the Volcani Center, Bet-Dagan, Israel. victor@volcani.agri.gov.il 2 Dept. of Industrial Engineering, Ben-Gurion University of the Negev, Israel 3 Agricultural and Biological Engineering, University of Florida, Gainesville, FL, 110570. Abstract. In this work we develop a real time system that recognizes occluded green apples within a tree canopy using infra-red and color images in order to achieve automated harvesting. Infra-red provides clues regarding the physical structure and location of the apples based on their temperature (leaves accumulate less heat and radiate faster than apples), while color images provide evidence of circular shape. Initially the optimal registration parameters are obtained using maximization of mutual information. Haar features are then applied separately to color and infra-red images through a process called Boosting, to detect apples from the background. A contribution reported in this work, is the voting scheme added to the output of the RGB Haar detector which reduces false alarms without affecting the recognition rate. The resulting classifiers alone can partially recognize the on-trees apples however when combined together the recognition accuracy is increased. Keywords: Mutual information, multi-modal registration, sensor fusion, Haar detector, apple detection. Introduction In the last few years, object recognition algorithms are focusing on the efficient detection of objects in natural scenes. A system is developed to recognize in real-time partially occluded apples regardless of position, scale, shadow pattern and illumination within a tree canopy. The work is motivated by the fact that labor for orchard tasks constitutes the largest expense (Jiménez et al., 2000), and hence there is a need to develop autonomous robotic fruit picking systems. Here we address the first step in such a system by tackling the problem of on tree green apple detection using real-time machine vision algorithms. The complexity of the task involves the successful discrimination of “green” apples within scenes of “green leaves”, shadow patterns, branches and other objects found in natural tree canopies. Color and edges are features highly dependent on illumination while texture is highly sensitive to the proximity (scale) of the object. An excellent review regarding apple recognition systems was presented in (Jiménez et al., 2000b). The concept of background modeling using Gaussian mixture color distributions in RGB images was used in Tabb et al., (2006). This algorithm detected 85 to 96 percent of both red and yellow apples assuming a uniform background in an artificial environment. Color distribution models for fruit, leaf and background classes were used in Annamalai et al., (2003) in a citrus fruit counting algorithm. In Stanjnko et al., (2004) pixel thermal values were mapped to RGB values and detected using the normalized difference index. However the efficiency of the algorithm was affected by the apple’s position on the tree and degree of sunlight. In Sapina (2001), textural features extracted from the gray level co-occurrence matrix were used to discriminate between warm objects and their background in thermal images. On the same vein, a