A Multimodal Approach to Feature Extraction for Image and Signal Learning Problems Damian R. Eads a,c , Steven J. Williams a , James Theiler a , Reid Porter b , Neal R. Harvey a , Simon J. Perkins a , Steven P. Brumby a , and Nancy A. David a a Space and Remote Sensing Sciences Group c Department of Computer Science b Space Data Systems Group Rochester Institute of Technology Los Alamos National Laboratory 102 Lomb Memorial Drive Los Alamos, NM, USA Rochester, NY, USA ABSTRACT We present zeus, an algorithm for extracting features from images and time series signals. Zeus is designed to solve a variety of machine learning problems including time series forecasting, signal classification, image and pixel classification of multispectral and panchromatic imagery. An evolutionary approach is used to extract features from a near-infinite space of possible combinations of nonlinear operators. Each problem type (i.e. signal or image, regression or classification, multiclass or binary) has its own set of primitive operators. We employ fairly generic operators, but note that the choice of which operators to use provides an opportunity to consult with a domain expert. Each feature is produced from a composition of some subset of these primitive operators. The fitness for an evolved set of features is given by the performance of a back-end classifier (or regressor) on training data. We demonstrate our multimodal approach to feature extraction on a variety of problems in remote sensing. The performance of this algorithm will be compared to standard approaches, and the relative benefit of various aspects of the algorithm will be investigated. Keywords: machine learning, support vector machines, genetic programming, remote sensing, image processing, time series analysis, lightning, classification, regression, automated feature extraction 1. INTRODUCTION In a previous publication, 1 we introduced zeus as a pattern recognition tool for time-series signals. Our initial interest was with the classification of lightning strikes (as measured with a high-speed radio-frequency receiver on the Fort´ e satellite 2 ), but we have since extended the zeus software in two directions. One extension permits zeus to be used for images as well as time-series. Also, where the earlier zeus was based on a model in which the entire time-series was a single (high-dimensional) sample point, the new zeus can be used in a mode that identifies each time point (or, for the images, each pixel) as a separate sample. This allows zeus to be used for segmenting different epochs within a long time-series signal, or for producing a pixel-by-pixel classifications within an image. Thus, zeus is designed for use in four separate modes, as described in Table 1: time series forecasting, time series classification, image classification, pixel-by-pixel classification within an image. Although the problems that characterize these modes are quite different in character, many of the same tools are used in their solution, and zeus provides a framework for incorporating those tools in a way the permits them to be used for a wide range of applications. Zeus is part of the Intelligent Searching of Images and Signals 3 project at Los Alamos, and follows other pattern recognition software that has been developed as part of that project, including genie 4–6 and afreet. 7 The previous generation of zeus was implemented in C++, but the more recent implementation of zeus has been rewritten with the kernel and interface in Java, and most of the mathematical processing in matlab . We found that the higher-level language provides the ability to more rapidly prototype new modules and capabilities. Zeus uses the mfitsio 8 package for reading and writing data, and the OSU SVM package 9 for the support vector machine back-end. For more information on the Zeus project, see http://www.zeus.lanl.gov. 10 Send correspondence to Damian Ryan Eads – E-mail: eads@lanl.gov, Telephone: (505) 667-9575, Address: Space and Remote Sensing Sciences Group, Los Alamos National Laboratory, MS D436, Los Alamos, NM, USA