J. Phys. B: At. Mol. Opt. Phys. 30 (1997) 309–318. Printed in the UK PII: S0953-4075(97)73442-4 Nonlinear coupling between rotation and internal vibration in simple molecular systems Yibing Li and Estela Blaisten-Barojas Institute for Computational Sciences and Informatics, George Mason University, Fairfax, VA 22030, USA Received 10 April 1996, in final form 28 August 1996 Abstract. A Hamiltonian is proposed to describe the coupling between vibration and rotation in a molecular system. The time behaviour of such a system exhibits both regular and irregular motion for different energies and different values of the coupling parameter. There is a dramatic manifestation of the transition between the two types of dynamics that define parametric regions of regular and irregular behaviour. The Lyapunov exponent, phase portraits, Poincar´ e sections and power spectra are calculated. The computer simulations show that the vibrational anharmonicity favours the regular behaviour of the system. 1. Introduction The study of complex irregular dynamics may be said to have started with the work of the French mathematician J H Poincar´ e at about the turn of the century. Although qualitative dynamics has been known to exist for a long time, its importance for a broad variety of molecular applications began to be appreciated only within the last decade. There are two main lines of research in this rapidly developing field: investigation of dissipative systems and research on Hamiltonian chaos. Concurrently, there has been enormous interest both within the mathematical community and among engineers and scientists to apply simple Hamiltonians to the description of a ‘few’ selected degrees of freedom relevant in complex systems. The field continues to develop rapidly, and its applications in material sciences are growing because of the fascinating changes in the time evolution of molecular systems [1–3]. Several Hamiltonian systems with two degrees of freedom are known to exhibit a transition from regular to chaotic motion as the energy of the system is increased. This is the case of the well studied two-dimensional Henon and Heiles system [4]. However, fewer studies have been undertaken which carefully examine the role played by the coupling parameters in a Hamiltonian system while the energy is kept constant. To be able to control these coupling parameters is one of the novel approaches of this technology. In this paper we present a two-degrees-of-freedom Hamiltonian system which is suitable for the description of molecular vibration–rotation processes. In this model we assume that the two relevant variables of the motion are represented by a stretching vibration coupled with the rotation around an axis perpendicular to the vibrational displacement that passes through the oscillator equilibrium position. The coupling energy is V 0 (1 − cos nq) [5] where the potential barrier V 0 is a function of the amplitude of vibration r and q is the angle of rotation. In this paper, we considered the case of n = 4, which is representative 0953-4075/97/020309+10$19.50 c  1997 IOP Publishing Ltd 309