JOURNAL OF LIGHTWAVE TECHNOLOGY, VOL. 28, NO. 19, OCTOBER 1, 2010 2851 Modified Smooth Transition Method for Determination of Complex Modes in Multilayer Waveguide Structures Ali Khalatpour, Jianwei Mu, Kaveh Moussakhani, and Wei-Ping Huang, Senior Member, IEEE Abstract—The smooth transition method for determination of complex leaky modes in multilayer waveguides is revisited. The mapping between the mode solutions of the initial close waveguide and that of the final open waveguide is discussed and established. It is suggested that a more suitable initial guess should be the real guided modes and the imaginary evanescent modes in a close waveguide with perfect reflecting boundary conditions. Further, the artificial boundary should be positioned right at the two farthest interfaces of the multilayer stacks adjacent to the outer cladding layers. The accuracy, efficiency, and robustness of the new algorithm are validated through calculating the leaky modes in a multilayer waveguide structure and three-layer slab waveguide structures. Index Terms—Complex modes, leaky modes, multilayer planar waveguide. I. INTRODUCTION M ULTILAYER planar waveguides play crucial roles in photonic devices and integrated circuits. The prerequi- site of the simulation, analysis, design and optimization of the multilayer planar waveguide structure is to determine the com- plete set of guided modes and radiation modes. In principle, the radiation modes have to be considered in many cases to expand the arbitrary fields of the open waveguide. In practice, however, the continuum nature of the radiation modes makes them hard to use [1]. The discrete leaky modes, on the other hand, may ap- proximately represent a cluster of radiation modes under some circumstance. The leaky modes are unbounded by nature and hence lack the usual characteristics of normal guided modes in terms of normalization and orthogonality. Yet they can be uti- lized in mode expansion together with guided modes to signif- icantly simplify the analysis of mode coupling problems in op- tical waveguides [2]. An array of numerical algorithms has been proposed to ob- tain the complex leaky modes. In the case of multilayer slab waveguides, the implicit characteristic is obtained Manuscript received April 12, 2010; revised July 03, 2010; accepted Au- gust 02, 2010. Date of publication August 12, 2010; date of current version September 20, 2010. This work was supported in part by a NSERC Research Grant. The work of J. Mu was supported in part by a NSERC Alexander Graham Bell Canada Graduate Scholarship (CGS). The authors are with the Department of Electrical and Computer Engi- neering, McMaster University, Hamilton, ON L8S 4K1, Canada (e-mail: muj2@mcmaster.ca). Color versions of one or more of the figures in this paper are available online at http://ieeexplore.ieee.org. Digital Object Identifier 10.1109/JLT.2010.2066548 from the transfer matrix method [3], [4], where is the propaga- tion constant. However, solving the above transcendental equa- tion for complex propagation constants is by no means an easy task. The problem is complicated for several reasons: 1) is a multi-valued equation and attention has to be paid for de- termining the signs according to the direction of power flow in the outer layers; 2) the propagation constants are complex so the root searching has to be conducted in the complex plane. The first problem can be circumvented by means of complex variable transformation so that the multi-valued equation can be mapped to a single valued one [5]. The second issue, i.e., locating the complex roots, is far more challenging since the generic root searching technique is critically dependent on the initial guess of the complex solutions, which is normally not known a priori. The situation is even more acute for applications in which a large number of bounded and leaky modes need to be captured (e.g., for the mode matching method or the com- plex coupled mode theory). Argument principle method (APM) is effective in finding all the complex roots; however, contour in- tegration in complex plane requires familiarity of the waveguide structure and is difficult for implementation [6], [7]. The reflec- tion pole method (RPM) detects the phase change of the denom- inator of the reflection coefficient and can predict the number of leaky modes in a chosen range, but is not efficient and accurate in practice as the results have to be obtained through curve fit- ting each time [8]. Recently a simple and effective numerical approach known as smooth transition method (STM) has been proposed for com- plex mode calculation [9]. A controllable artificial boundary is introduced outside of the multilayer stacks and the real guided mode solution for the close waveguide is used as the initial guess for the complex root searching. The root locus moves from real axis to complex plane as the boundary gradually changes from close to open. Although the STM appears to be simple and effec- tive in locating the complex roots associated with leaky modes, it also has its problems and limitations. The number of real ini- tial solutions and the mode spectral separation of the close wave- guide are determined by and dependent on the positions of the artificial perfectly reflecting boundaries. It is still not clear how to choose the proper positions of the perfect reflecting bound- aries and what will be the optimum initial conditions to start with. Further, for higher order leaky modes with relatively large leakage loss, the real initial guesses of the close waveguides used in the conventional STM are not effective and may even fail. Lastly, since the number of initial solutions does not match exactly the number of final solutions, conventional STM may either miss some solutions or generate spurious solutions. 0733-8724/$26.00 © 2010 IEEE