Automated Annotation of Coral Reef Survey Images Oscar Beijbom † Peter J. Edmunds * David I. Kline ‡ B. Greg Mitchell ‡ David Kriegman † {obeijbom, dkline, gmitchell, kriegman}@ucsd.edu, peter.edmunds@csun.edu { † Department of Computer Science and Engineering, ‡ Scripps Institution of Oceanography}, University of California, San Diego. * Department of Biology, California State University Northridge. Abstract With the proliferation of digital cameras and automatic acquisition systems, scientists can acquire vast numbers of images for quantitative analysis. However, much image analysis is conducted manually, which is both time consum- ing and prone to error. As a result, valuable scientific data from many domains sit dormant in image libraries awaiting annotation. This work addresses one such domain: coral reef coverage estimation. In this setting, the goal, as de- fined by coral reef ecologists, is to determine the percent- age of the reef surface covered by rock, sand, algae, and corals; it is often desirable to resolve these taxa at the genus level or below. This is challenging since the data ex- hibit significant within class variation, the borders between classes are complex, and the viewpoints and image quality vary. We introduce Moorea Labeled Corals, a large multi- year dataset with 400,000 expert annotations, to the com- puter vision community, and argue that this type of ecologi- cal data provides an excellent opportunity for performance benchmarking. We also propose a novel algorithm using texture and color descriptors over multiple scales that out- performs commonly used techniques from the texture clas- sification literature. We show that the proposed algorithm accurately estimates coral coverage across locations and years, thereby taking a significant step towards reliable au- tomated coral reef image annotation. 1. Introduction In many scientific disciplines experts routinely analyze large quantities of image data. However, not only has the capacity for acquiring digital images vastly outpaced the resources to manually annotate those images, but there are also issues with lack of consistency and objectivity in hu- man labeling [18, 8]. One such domain is coral reef ecol- ogy, which is particularly important given the crucial eco- logical roles of coral reefs, and their current state of de- cline in health and abundance [26]. To understand this de- Figure 1. Moorea Labeled Corals example images: Top row im- ages are from the Outer 10m reef habitat, bottom from the fringing reef. Superimposed on each image is a subset of the ground truth annotations (smaller symbols) and the estimated classifications by the proposed algorithm (larger symbols). Circles represent coral genera Acropora, Pavona, Montipora, Pocillopora, Porites and tri- angles are non-coral substrates, Crustose Coralline Algae, Turf al- gae, Macroalgae and Sand. Note the organic borders between the substrates and complex class morphologies. The white transect line going through the images is part of the sampling methodol- ogy, as is the metal frame seen along the edges. This figure is best viewed in color. velopment, ecologists need accurate and large-scale coral reef coverage data. As satellite images are ineffective for this purpose [14] and low altitude photography suf- fers from problems such as surface effects, coral ecologists commonly do in situ studies. Recent innovations in im- age acquisition techniques, such as autonomous underwa- ter vehicles [24] and towed diver sleds [15] have greatly