Nonlinear dynamics of higher-order solitons near the oscillatory instability threshold Kazimir Y. Kolossovski and Alexander V. Buryak School of Mathematics and Statistics, University of New South Wales, Australian Defence Force Academy, Canberra 2600, Australia Dmitry V. Skryabin Department of Physics and Applied Physics, University of Strathclyde, 107 Rottenrow, Glasgow G4 0NG, United Kingdom and Department of Physics, University of Bath, Bath BA2 7AY, United Kingdom Rowland A. Sammut School of Mathematics and Statistics, University College Australian Defence Force Academy, Canberra ACT 2600 Australia Received 19 March 2001; published 23 October 2001 Nonlinear theory describing the dynamics of solitons in the vicinity of oscillatory instability threshold with a low frequency offset is developed. The theory is tested on the example of parametric degenerate four-wave mixing. All major predictions of our theory are in agreement with the results of direct numerical modeling. This includes the position of oscillatory instability threshold, instability rates, and various instability develop- ment scenarios. DOI: 10.1103/PhysRevE.64.056612 PACS numbers: 42.65.Tg, 42.81.Dp, 05.45.Yv I. INTRODUCTION Among the most fascinating objects of nonlinear science are solitons—self-guided beams and pulses. One of the rea- sons for the current interest in optical solitons is due to the possibility of all-optical switching and controlling light by light see, e.g., 1. Continuous growth of theoretical knowl- edge for existence, effective generation, and stability of dif- ferent types of solitary waves is backed by a variety of suc- cessful experiments 2. Stability of solitons is one of the paramount questions see e.g., 3 and references therein. The majority of theo- retical results related to the dynamics of solitary waves were obtained via linear spectral stability analysis. This describes well only the initial stages of perturbed soliton evolution and leaves unresolved its subsequent dynamical behavior. During the last decade several model equations, like generalized nonlinear Schro ¨ dinger equation NLS4,5 and three-wave mixing 6,7, were used to develop theory describing longer- term nonlinear dynamics of solitons near the Vakhitov- Kolokolov VK instability threshold 8. This theory suc- cessfully explains persistent oscillations, decay and collapse of the solitary waves, and transitions between these scenarios 4. VK instability being a particular example of the instabil- ity generated by purely real eigenvalues in the soliton spec- trum, is a typical first instability for the fundamental node- less solitary solution. Higher-order excited-state solutions are also of signifi- cant fundamental and practical interest. Recently, wide inter- est in such solitons has been aroused by the discovery of several classes of stable higher-order solitons in different nonlinear media 9,10. The most typical scenario of insta- bility of the higher-order states is, however, instability due to complex eigenvalues. One of the first examples of complex eigenvalues in the linear spectrum of a Hamiltonian system is associated with the antisymmetric mode of nonlinear pla- nar waveguide, Refs. 11. For more recent examples of os- cillatory instability see Refs. 12. Analytical treatment of such instabilities is much more involved and one of the first steps in this direction was made in 13, where the general linear asymptotic stability analysis capable of capturing complex eigenvalues has been developed, but has not been backed up with any physical example. The approach used in 13 is based on the assumption that oscillatory and VK in- stabilities happen sufficiently close to each other in the vi- cinity of the codimension two point where four correspond- ing eigenvalues merge at zero and the governing eigenvectors coincide. On the other hand, recent numerical studies of the spectral properties of the optical solitons de- generate four-wave mixing FWM10 have revealed the existence of exactly such a point for one of the higher-order soliton families, suggesting that this situation may be much more typical for complex nonlinear evolutional models than previously thought. The goal of this work is a detailed study of this FWM example and nonlinear generalization of the linear theory of Ref. 13. We derive a nonlinear ordinary differential equation ODE model that provides a valuable insight into longer-term instability-induced dynamics of higher-order solitons allowing us to classify different insta- bility development scenarios. All major predictions of our analytic results are in full agreement with direct numerical simulations. II. DEGENERATE FOUR-WAVE MIXING MODEL We intend to demonstrate all major steps of the derivation of our nonlinear dynamics theory in the example of a specific physical model representing a degenerate case of parametric FWM in the presence of self- and cross-phase modulation. The equations for this model are i  U  Z +  2 U  X 2 - U +N 1 + 1 3 U * 2 W=0, 1 i   W  Z +  2 W  X 2 -  3  +  W+N 2 + 1 9 U 3 =0, PHYSICAL REVIEW E, VOLUME 64, 056612 1063-651X/2001/645/05661211/$20.00 ©2001 The American Physical Society 64 056612-1