A Novel Approach for Designing Cosine-Modulated Transceivers Pilar Martín-Martín, 1 Fernando Cruz-Roldán, 1 and Tapio Saramäki 2 Dpto. Teora de la Seæal y Comunicaciones 1 Universidad de AlcalÆ AlcalÆ de Henares, Madrid, Spain e-mail: p.martin@uah.es, fernando.cruz@uah.es Institute of Signal Processing 2 Tampere University of Technology P. O. Box 553, FIN-33101 Tampere, Finland e-mail: ts@cs.tut.fi Abstract This paper introduces a novel approach for designing critically sampled cosine-modulated transceivers (CMTs) for communications applications. The key idea in this ap- proach is to design the prototype filter in such a manner that it provides a properly optimized tradeoff between the inter-symbol and inter-channel interferences for CMTs without directly concentrating on designing the prototype filter according to any predetermined criteria. In order to generate a very fast technique for testing this approach, the windowing technique is used for optimizing the proto- type filter and the window function is a generalized Blackman window in the sense that this window function has the fourth additional term. Using this windowing ap- proach results in a very fast unconstrained optimization problem with only three unknowns involved for optimally determining the above-mentioned tradeoff for a CMT, in- dependently of the order of the prototype filter and the number of channel of the CMT. Several exampled are in- cluded showing that CMTs with very good properties can be achieved very quickly by using the proposed tech- nique. 1. Introduction THE DISCRETE MULTITONE TECHNIQUE can be ana- lyzed from the point of view of a critically sampled M- channel transceiver system as shown in Fig. 1, where the filters in the transmitting and the receiving filter banks are IDFT and DFT filters, respectively. It is well known that the selectivity of the transmitting and receiving filters in such an overall system is rather limited due to the fact that their stop-band attenuation is only approximately 13 dB. The main advantages of this multi-carrier system are a low complexity and the ability to combat the main inter- ference, namely, the inter-symbol interference (ISI), by adding a cyclic prefix [1, 2]. In order to get higher at- tenuation for the channel filters in Fig. 1, the systems based on the use of the transceivers with overlapping- block transmission are often used [1−3]. In this case, the discrete wavelet multi-tone technique (DWMT) replaces This work was supported in parts by Universidad de AlcalÆ and the Academy of Finland, project No. 44876 [Finnish centre of Excellence program (2000-2005)]. The work was performed while the first author was visiting the Institute of Signal Proc- essing, Tampere University of Technology. the DFT by a discrete wavelet transform in both the transmitting and receiving banks. A very attractive way for implementing this DWMT type of filter banks is to use cosine-modulated transceivers (CMTs) [2]. In this case, both higher stop-band attenuations and lower side- lobes for the filters in both banks of Fig. 1 can be achieved when compared to the use of the complex modulation. This is because when using the cosine modu- lation, the sub-filter orders are usually selected to be N = 2KM−1, where M is the number of sub-channels and K is the overlapping factor, whereas the filter orders is only N = M−1 for the complex modulation. Furthermore, CMTs can be designed quite quickly because all the fil- ters are derived from a single prototype filter, and there exist computationally efficient structures for their imple- mentation. Figure 1. Critically sampled M-channel transceiver over a noisy channel. The main problem in CMTs is that they suffer from a severe ISI and the equalization techniques are more com- plicated. In this paper, a novel approach is proposed for designing prototype filters for generating CMTs in such a manner that the ISI is considerably decreased when com- pared to other existing approaches [4]. Generating this approach is motivated by the follow- ing two facts. First, perfect-reconstruction (PR) CMFs are not worth using since in real communications environ- ments this property is not needed due to the errors caused by the transmission channel. Therefore, it is more benefi- cial to use nearly PR (NPR) CMTs as long as the errors caused by the use of NPR CMTs are significantly smaller than those caused by the transmission channel. By prop- erly exploiting this fact results either in CMTs with better filter banks performances or in CMTs meeting the same performance with a reduced arithmetic complexity and a lower overall filter bank delay. Second, and most impor- tantly, most of the existing proposed approaches concen- trate on designing the prototype filters in such a manner