Adaptive clustering for hyperspectral sounder data compression ∗ I. Gladkova † , L. Roytman † , M. Goldberg ‡ † City College of New York, NOAA/CREST, 138th Street and Convent Avenue, New York, NY 10031 ‡ NOAA/NESDIS, Office of Research & Applications, 1335 E/W Highway, Silver Spring, MD 20910 ABSTRACT In this paper, which is part of an ongoing sequence of papers devoted to the subject of efficient noise-tolerant lossless compression of satellite data for transmission, we describe an algorithm for this purpose which effectively addresses the above criteria. Our algorithm exhibit the potential to achieve noise-tolerant compression ratios averaging 3.2 : 1. An earlier approach, which we presented at Third GOES-R User Conference 6 in May of 2004, was the first such method to break the 3 to 1 compression barrier for this class of data. Keywords: compression, subspaces, clustering. 1. INTRODUCTION The compression technique described in this paper arose from requirements imposed by several considerations which are desirable in the processing of data from NASA/NOAA’s environmental satellites. The most important of these are speed, efficient memory utilization, and lossless compression. The motivation for a need for good compression algorithms is obvious. One example: data flow from the current NASA Aqua satellite can reach 89 GB/day. Any algorithm which exhibits good performance in the above sense is a good candidate for on board processing of data prior to transmission. In addition to the above characteristics, such an algorithm should tolerate acceptably the inevitable presence of noise introduced during transmission. With these criteria in mid, our algorithms are currently in the process of being tested as suitable interfaces with NOAA’s existing data retrieval packages. When this testing is completed, we will report on the results. This paper should be regarded as one of a sequence of papers, some in current development or preparation, devoted to the development and description of algorithms for the efficient transmission of satellite data. A central principle of our work in this area is the attempt to cluster the data-points along or close to low-dimensional hyperplanes, since the actual satellite data we have been working with supports the hypothesis that such clustering in fact is present. A key problem in such an approach is the identification of such hyperplanes, which turns out to be an interesting and challenging problem. A key requirement of any practical treatment of this problem is speed, because of current on board processing limitations. 2. DATA Our group is working on compression techniques that are suitable for the next-generation NOAA/NESDIS Geostation- ary Operational Environmental Satellite (GOES) instruments. We are using current spacecraft to simulate data from the upcoming GOES-R instrument and focusing on Aqua Spacecraft’s AIRS instrument in our case study. ∗ Sponsored by NOAA/NESDIS under Tim Schmit (ORA), Roger Heymann (OSD) HES Compression Group Invited Paper Satellite Data Compression, Communications, and Archiving, edited by Bormin Huang, Roger W. Heymann, Charles C. Wang, Proc. of SPIE Vol. 5889 (SPIE, Bellingham, WA, 2005) · 0277-786X/05/$15 · doi: 10.1117/12.618085 Proc. of SPIE 588907-1