Journal of Engineering Sciences, Assiut University, Vol. 37, No. 3, pp. 653-667 ,May 2009. 653 ADAPTIVE APPROACH FOR BLIND SOURCE SEPARATION OF NONLINEAR MIXING SIGNALS Usama Sayed Mohammed Department of Electrical Engineering, Faculty of Engineering, Assiut University, Assiut, Egypt E-mail: Usama@aun.edu.eg Hany Saber Department of Electrical Engineering, Faculty of Eng., South Valley University, Aswan, Egypt E-mail: hany@svu.edu.eg (Received December 20, 2008 Accepted May 9, 2009) In this paper, a new technique to solve the nonlinear blind source separation problem (NBSS) is introduced. The method is based on the concept of reducing the high frequency component of the nonlinear mixed signal by dividing the mixed signal into blocks in the time domain, with any arbitrary size. To remove the distortion of the nonlinear function, the discreet cosine transform (DCT) is applied on each block. By adaptively adjusting the size of the DCT block of data, the highly correlated subblocks, can be estimated, then the correlation between the highly correlated sub-blocks can be reduced. To complete the separation process, the linear blind source separation (BSS) algorithm based on the wavelet transform is used to reduced the correlation between the highly correlated DCT subblock. Performed computer simulations have shown the effectiveness of the idea, even in presence of strong nonlinearities and synthetic mixture of real world data (like speech and image signals). KEYWORDS: nonlinear blind source separation, discreet casein transform, linearization, post nonlinear mixing and independent component analysis (ICA). 1. INTRODUCTION The problem of blind source separation (BSS) consists on the recovery of independent sources from their mixture. This is important in several applications like speech enhancement, telecommunication, biomedical signal processing, etc. Most of the work on BSS mainly addresses the cases of instantaneous linear mixture [1-5]. Let A a real or complex rectangular (n×m; n≥m) matrix, the data model for linear mixture can be expressed as AS(t) X(t)  (1) Where S(t) represents the statistically independent sources array while X(t) is the array containing the observed signals. For real world situation, however, the basic linear mixing model in equation (1) is too simple for describing the observed data. In many applications such as the nonlinear characteristic introduced by preamplifiers of receiving sensors, we can consider a nonlinear mixing. So a nonlinear mixing is more realistic and accurate than linear model. For instantaneous mixtures, a general nonlinear data model can have the form