Abstract— In this paper, a hybrid optimization for the design infinite impulse response (IIR) filter employing particle swarm optimization (PSO) and artificial bee colony (ABC) algorithm is presented. The design problem is formulated as a mean square error between desired response and ideal response. For the stability of filter, a lattice structure in design is utilized. A comparative study of performance of PSO and its different variants is also made which shows improved efficiency of hybrid method in the field of IIR filter design. Index Terms—IIR, Hybrid, PSO, Quantum Particle Swarm Optimization (QPSO), Mean Square Error (MSE) I. INTRODUCTION SIGNAL processing has been embryonic out as an innovative rapid developing stream of modern engineering. In digital signal processing, digital filter is a program, which is executed on dedicated hardware and performs arithmetic operation on sample of discrete time signal. Based on the impulse response, the digital filters are divided into two types: finite impulse response filter (FIR) and IIR filter. For same specifications, IIR filter gives better performance as compared to IIR filter in term of computational complexity. Commonly, two techniques for the design of IIR filters are used: analog filter transformation (conventional method) and computer aided design method [1]. In conventional method, there is no control on transition band, and order of filter depends on the permissible attenuation at edge frequencies of passband and stopband and cutoff frequency. Hence, for meeting the superior performance, higher order filter is required. To overcome these issues, the computer aided design techniques such as pole and zero placement method, have been developed [1]. These methods were closed form or iterative in nature. The closed form method uses some predefined mathematical relation such as Lagrange multiplier for designing digital filter [2]. In iterative based method, cycles of iteration were computed to optimize the objective function [3]. Recently, evolutionary algorithms for global optimization /search has been developed and employed in various engineering optimization problems. Since, IIR filter design problem is nonlinear in nature therefore, it has multi-model error surface, and so linear optimization techniques are not suitable for design optimized IIR filter. In recent decades, The authors are with the PDPM Indian Institute of Information Technology, Design and Manufacturing, Jabalpur, MP, 482005 India; (e-mail- nikhil.agrawal@iiitdmj.ac.in, anilk@iiitdmj.ac.in, and varunb@iiitdmj.ac.in). Meta heuristic global search /optimization techniques have quit significant development and successfully applied on various optimization problems [4-7]. However, these techniques do not guarantee the optimal solution to the problem, but have the capability to explore globally and not to fix in local minima. Several researchers have used genetic algorithm (GA) and its modified versions in various applications such as training of neural network, designing of optimal digital FIR and IIR filter, design of adaptive FIR filter [3, 8-11]. Exploring for better solution, single link search method like, simulated annealing (SA) has also proposed for optimizing IIR filter [12, 13]. The genetic algorithms has some limitation like it is non-deterministic, result depends highly on the initial population. If the diversity method in GA does not work properly, the algorithm often too soon converge into local optima. Similarly in SA, there is always a trade-off among the value optimized and the time required to compute them. The rest of the paper is arranged as follows. In Section II, design formulation of IIR filter is presented. In Section III, a brief discusses on the swarm optimization algorithms is presented. Section IV explains the concept of hybrid PSO. In Section V, simulation results for low pass (LP) and high pass (HP) filters and a discussion on the results are given. Finally, Section VI concludes the paper. II. DESIGNING OF IIR FILTER A. Designing of IIR filter In this Section, formulation of design problem for IIR filter is explained. Any discrete system transfer function is represented by Eq. (1), where a k is numerator coefficients and b k is denominator coefficients [1]. 0 1 () () () 1 m k k k n k k k a z Yz Hz Xz b z            (1) where, m and n are equal to the order of filter. Eq. (1) is further represented as: 1 1 2 0 2 1 2 1 2 3 () 1 m m n a a z a z a z Hz b z b z b z                      (2) If z = e jω , Eq. (2) is modified as Hybrid method based optimized design of digital IIR filter N. Agrawal, A. Kumar, and V. Bajaj, International Conference on Communication and Signal Processing, April 2-4, 2015, India ISBN 978-1-4799-8080-2 Adhiparasakthi Engineering College, Melmaruvathur 1568