AN EFFICIENT APPROACH FOR DESIGNING NEARLY PERFECT- RECONSTRUCTION COSINE-MODULATED AND MODIFIED DFT FILTER BANKS Tapio Saramäki and Robert Bregovi Signal Processing Laboratory Tampere University of Technology P. O. Box 553, FIN-33101 Tampere, Finland e-mail: ts@cs.tut.fi and bregovic@cs.tut.fi ABSTRACT Efficient two-step algorithms are described for optimizing the stopband response of the prototype filter for cosine-modulated and modified DFT filter banks either in the minimax or in the least-mean-square sense subject to the maximum allowable aliasing and amplitude errors. The first step involves finding a good start-up solution using a simple technique. This solution is improved in the second step by using nonlinear optimization. Several examples are included illustrating the flexibility of the proposed approach for making compromises between the re- quired filter lengths and the aliasing and amplitude errors. These examples show that by allowing very small amplitude and aliasing errors, the stopband performance of the resulting filter bank is significantly improved compared to the corresponding perfect-reconstruction filter bank. Alternatively, the filter orders and, consequently, the overall delay can be significantly reduced to achieve practically the same performance. 1. INTRODUCTION Among different classes of M-channel critically sampled filter banks, cosine-modulated [1]−[9] and modified DFT [10]−[11] filter banks have become very popular in many applications due to the following reasons. First, these banks can be generated using a single prototype filter by exploiting a proper transfor- mation, making the overall implementation effective. Second, the overall synthesis can concentrate on optimizing only the prototype filter. This paper concentrates on designing cosine- modulated filter banks, but as has been pointed out in [11], the same prototype filter with a proper scaling can be used for both filter types mentioned above. For designing the prototype filter different strategies can be applied. The design can be performed using constrained mini- mization [5], iterative methods [6], lattice factorizations [2], [3], [4] as well as by applying some other synthesis schemes [7], [8]. Some of these methods result in perfect-reconstruction (PR) fil- ter banks whereas some in nearly PR filter banks. For practical applications with lossy channel coding and quantization, the PR property is desirable but not necessary. In this case, the distortion caused by aliasing and amplitude errors to the signal is allowed provided that they are smaller than that caused by coding. Therefore, it is worth trying to release the PR condition with the ultimate goal being to achieve better filter bank properties. This work was supported by the Academy of Finland, project No. 44876 (Finnish centre of Excellence program (2000-2005)). This paper describes an efficient two-step approach for syn- thesizing prototype filters for nearly PR filter banks. In the first step, a proper prototype filter for a PR filter bank is generated using a systematic multi-step procedure described in [12]. In the second step, this filter is used as a start-up solution for solving the given constrained optimization problem. The optimization is carried out by using the second algorithm of Dutta and Vidyasa- gar [9], [13]. Several examples are included illustrating that by allowing small amplitude and aliasing errors, the filter bank performance can be significantly improved. Alternatively, the filter orders and the overall delay caused by the filter bank to the signal can be considerably reduced. This is very important in communication applications. F 1 (z) H 1 (z) F 0 (z) ↑Μ ↑Μ ↓Μ H 0 (z) ↓Μ + y(n) x(n) HM-1(z) FM-1(z) ↑Μ ↓Μ + Figure 1. M-channel maximally decimated filter bank. 2. COSINE-MODULATED FILTER BANKS A general M-channel critically sampled filter bank is shown in Figure 1 [1]. For this system the input-output relation in the z- domain is expressible as ( ) ∑ − = − + = 1 1 2 0 ) ( ) ( ) ( ) ( M l M l j l ze X z T z X z T z Y π , (1a) where ∑ − = = 1 0 0 ) ( ) ( 1 ) ( M k k k z H z F M z T (1b) and ( ) ∑ − = − = 1 0 2 ) ( 1 ) ( M k M l j k k l ze H z F M z T π (1c) for l = 1, 2, …, M−1. Here, T0(z) is called the distortion transfer function and determines the distortion caused by the overall system for the unaliased component X(z) of the input signal. The remaining transfer functions Tl(z) for l = 1, 2, …, M−1 are called the alias transfer functions and determine how well the aliased components X(ze −j2πl / M ) of the input signal are attenuated. For PR, it is required that T0(z) = z −N with N being an integer and Tl(z) = 0 for l = 1, 2, …, M−1. If these conditions are satis-