Content Addressable Memories in Reproducing Kernel Hilbert Spaces Erion Hasanbelliu, and Jose C. Principe, Member, IEEE Abstract--Content addressable memories (CAM) are one of the few technologies that provide the capability to store and retrieve information based on content. Even more useful is their ability to recall data from noisy or incomplete inputs. However, the input data dimension limits the amount of data that CAMs can store and successfully retrieve. We propose to increase the amount of information that can be stored by implementing CAMs in a reproducing kernel Hilbert space where the input dimension is practically infinite, effectively lifting this CAM limitation. We show the advantages of kernel CAMs over CAMs by comparing their performance in information retrieval, generalization, storage, and online learning. I. INTRODUCTION H UMAN memory is believed to be associative where events are linked to one another in such a way that the occurrence of an event, stimulus, triggers the emergence of another event, response. This association is strengthened through time by the constant trigger of the response via the stimulus event; a learning process that is known as Hebb's rule [1]. There are two main types of associative memory: autoassociative memory where a stored pattern which most closely resembles the stimulus pattern is retrieved, and heteroassociative memory where the retrieved pattern is the response of a stored stimulus that closely matches the input pattern. The theory of associative memories is well researched by some of today's most renowned research scientist like Kohonen and Anderson. A well known type of autoassociative memory is the Hopfield model [2], which is an unsupervised recurrent neural network. The Hopfield network computes its output recursively in time until it reaches a stable (attractor) point which is one of the stored patterns. Heteroassociative memory, on the other hand, is simpler and the output pattern is computed immediately from the stimulus pattern and the association matrix (memory) [3]. A CAM can be thought as a linear network being trained Manuscript received August 7, 2008. This work was supported in part by the National Science Foundation under Grant ECS-0601271 and in part by the Office of Naval Research under Grant NOOOI4-07-1-0698. Erion Hasanbelliu is with the Computational NeuroEngineering Laboratory, ECE Department, University of Florida, Gainesville, FL 32611 USA (phone: 352-392-2682; fax: 352-392-0044; e-mail: erioni@cne1.ufl.edu). Jose C. Principe is with the Computational NeuroEngineering Laboratory, ECE Department, University of Florida, Gainesville, FL 32611 USA (e-mail: principe@cne1.ufl.edu). with input output patterns very similarly to regression. In order to avoid cross-talk amongst the stored patterns, the stored patterns must be orthogonal. Since an N dimensional vector space has only N orthogonal directions, it is only possible to store without crosstalk N memories with N components. This becomes the most fundamental limitation of CAMs. The advent of Reproducing Kernel Hilbert Spaces (RKHS) in machine learning and signal processing brings a new perspective to CAMs since for most kernels (like the Gaussian) the dimensionality of the RKHS is infinite. This yields the promise of CAMS with infinite storage, and attracted our attention. The rest of the paper is organized as follows. The next section describes heteroassociative memories in more detail and a way to improve its shortcomings. The third section explains associative memories using kernel methods. The fourth section presents several experimental results of comparing the kernel CAM against the original CAM. And, the last section concludes the paper with some comments on this work. II. CONTENT ADDRESSABLE MEMORIES A. Heteroassociative memory Content addressable memories utilize Hebb's learning rule to associate a certain input state vector x with an output state vector d. The connections between the input and output patterns are stored in a matrix W, which is computed using the outer product rule W =1] . d.x T • The system is considered to have learned the association when whenever an input vector x is presented, the corresponding output vector d is retrieved. The output state vector is retrieved by multiplying the connection matrix with the input vector as follows: d, = W oX, = f d i ox/ oX, = f d i o(xi,x,)· (1) ;=1 ;=1 The heteroassociative memory works well when the input vectors are orthogonal (derivation is shown in appendix A). Thus, the system reconstructs the output pattern perfectly as long as the stored pairs are orthogonal. However, in general, the input vectors are not orthogonal, and as a result, there is potential for interference between the different association pairs. Another obvious limitation of associative memory is its limited capacity. The number of pairs that can be successfully stored in the connection matrix is dependent on the dimensionality of the state vectors; e.g., if the data 1-4244-2376-7/08/$20.00 ©2008 IEEE 9