1236 IEEE TRANSACTIONS ON CIRCUITS AND SYSTEMS—I: REGULAR PAPERS, VOL. 52, NO. 6, JUNE 2005 Neural Network Approach to Blind Signal Separation of Mono-Nonlinearly Mixed Sources W. L. Woo, Member, IEEE, and S. S. Dlay, Member, IEEE Abstract—A new result is developed for separating nonlinearly mixed signals in which the nonlinearity is characterized by a class of strictly monotonic continuously differentiable functions. The structure of the blind inverse system is explicitly derived within the framework of maximum likelihood estimation and the system culminates to a special architecture of the 3-layer perceptron neural network where the parameters in the first layer are inversely related to the output layer. The proposed approach exploits both the structural and signal constraints to search for the solution and assumes that the cumulants of the source signals are known a priori. A novel statistical algorithm based on the hy- bridization of the generalized gradient algorithm and metropolis algorithm has been derived for training the proposed perceptron which results in improved performance in terms of accuracy and convergence speed. Simulations and real-life experiment have also been conducted to verify the efficacy of the proposed scheme in separating the nonlinearly mixed signals Index Terms—Independent component analysis (ICA), neural networks, nonlinear distortion, nonlinear systems, signal reconstruction. I. INTRODUCTION I N RECENT TIMES, blind source separation (BSS) using in- dependent component analysis (ICA) has received attention because of its potential application in signal processing such as in speech recognition systems, telecommunications and med- ical signals processing. The goal of ICA is to recover indepen- dent sources given only sensor observations that are unknown linear mixtures of the unobserved independent source signals. Many if not complete theories regarding various aspects of the linear BSS have been established and confirmed experimentally [1], [2]. However, in general, and for many practical problems, mixed signals are more likely to be nonlinear or subject to some kind of nonlinear distortions due to sensory or environmental limitations. For example, in medical imaging systems, the mag- netic resonance and computed tomography are strongly affected by artifacts and nonlinearities which are generated from many independent sources [3]. In some signal and array processing applications, the components and sensor elements often exhibit nonlinear behavior at certain signaling conditions [4], [5]. In each case, there is an urgent need to be able to identify these nonlinearities and more importantly, ameliorate their effects in order to obtain a clear and accurate representation of the actual signals. Manuscript received August 28, 2003; revised June 29, 2004 and October 7, 2004. This paper was recommended by Associate Editor Y. Inouye. The authors are with the School of Electrical, Electronic and Computer Engi- neering, University of Newcastle upon Tyne, Newcastle upon Tyne, NE1 7RU, U.K. (e-mail: w.l.woo@ncl.ac.uk). Digital Object Identifier 10.1109/TCSI.2005.849122 II. NONLINEAR MIXING MODEL There have been increasing number of real-life applications involving the use of nonlinear models and as a result, the need to acquire nonlinear control algorithm has become a crucial part of the signal processing design. In this paper, a nonlinear model based on the theory of functional analysis [23] for modeling nonlinear mixtures is derived. The following lemma establishes the foundation upon which the model can be derived. Lemma 1 [23]: If an equation can be expressed in the form of where one of the func- tions (or ) is a continuous group operation for the , of an interval, then equation has a strictly monotonic continuous function if and only if the other function (or ) is also a continuous group operation. On applying Lemma 1, if two source signals (e.g., and ) are related to the function as and and the group operation is defined as where is simply an arbitrary scalar whose range is bounded within [ , 1] i.e., , then the group operation will assume the form of and the auto-associative operation Moreover, we may generalize the group operation to include three elements as follows: (1) where and . Following identical procedure as in (1), this can be ex- tended to include number of elements as . Now, if a nonlinear system with inputs and outputs can be defined as with where (set 1057-7122/$20.00 © 2005 IEEE