ADAPTATION IN THE QUINCUNX WAVELET FILTER BANK WITH APPLICATIONS IN IMAGE DENOISING Miroslav Vrankic 1 and Damir Sersic 2 1 Faculty of Engineering, University of Rijeka Vukovarska 58, HR-51000 Rijeka, CROATIA 2 Faculty of Electrical Engineering and Computing, University of Zagreb Unska 3, HR-10000 Zagreb, CROATIA e-mail: Miroslav.Vrankic@RITEH.hr, Damir.Sersic@FER.hr ABSTRACT In this paper, we present the realization of an adaptive shift invariant wavelet transform defined on the quin- cunx grid. The wavelet transform relies on the lifting scheme which enables us to easily introduce the adapta- tion by splitting the predict stage into two parts. The first part of the predict stage is fixed and guarantees the number of vanishing moments of the wavelet filter bank while the second part can adapt to the local properties of the analyzed image. In this paper, we explore the robustness of the generalized least squares adaptation algorithm to the noise present in the analyzed image. The denoising results obtained with the nonseparable adaptive wavelet transform have been compared with results obtained with both separable and nonseparable fixed wavelet transforms. Also, the empirical Wiener fil- tering in the wavelet domain has been used in order to further improve the denoising results. 1 INTRODUCTION In the field of image denoising, wavelets have been established as a very useful and effective tool. By using the wavelet transform, smooth regions of the image can be approximated very well with coarse approximation wavelet coefficients while most of the detail wavelet coefficients are zero or close to zero. On the other hand, edges (which contain most important information) as sharp transitions in the image are represented with high-valued detail coefficients. These properties of the wavelet transform guarantee the effectiveness of the image denoising procedure called wavelet shrinkage [1, 2]. When using the wavelet shrinkage to remove noise from the analyzed image, wavelet coefficients of the noisy image smaller than a given threshold are set to zero and the coefficients above the threshold are either left unchanged (hard thresholding) or reduced by the value of the threshold (soft thresholding). The higher the threshold, the more wavelet detail coefficients are being set to zero and the reconstructed image looks smoother. The good threshold is one that removes most of the noisy detail coefficients while still not oversmoothing the analyzed image, i.e. retaining the detail coefficients corresponding to the edges in the image. The basic motivation of our research was to im- prove the wavelet transform by making it locally adaptive to the properties of the analyzed image. Such an improved wavelet transform would approximate better important features of the image and the detail wavelet coefficients would remain dominated with the noise. Wavelet shrinkage based on such an improved wavelet transform is expected to give better denoising results. We have used the lifting scheme [3] in order to create an adaptive wavelet filter bank. Since the lifting scheme automatically guarantees perfect reconstruction of the resulting filter bank, it is very straightforward to introduce the adaptivity by simply making some of its basic building blocks adaptive. A number of adaptive wavelet transforms based on the lifting scheme have been proposed in the literature. In [4], the lifting scheme with the adaptive prediction step has been used to create an adaptive filter bank structure. Unfortunately, important proper- ties of the wavelet transform have been lost because of the fully adaptive prediction step. Claypoole et al [5] have proposed locally adaptive wavelet transform for image coding based on the lifting scheme where the order of the prediction filter is being changed in order to avoid prediction across the discontinuities. In the smooth parts of the image higher order prediction filters are being used (resulting in wavelets with more zero moments), while near the edges lower order prediction filters are being used. In [6] the update step in the lifting scheme has been made adaptive based on the local gradient information. The adaptation algorithm can be perfectly reproduced on the reconstruction side so that no additional book- keeping is required.