ISSN 1560-3547, Regular and Chaotic Dynamics, 2008, Vol. 13, No. 3, pp. 166–177. c  Pleiades Publishing, Ltd., 2008. RESEARCH ARTICLES Stability of Equilibrium Solutions of Hamiltonian Systems Under the Presence of a Single Resonance in the Non-Diagonalizable Case F. dos Santos 1* and C. Vidal 2** 1 Departamento de Matem´atica, Universidade Federal de Sergipe, Av. Marechal Rondon, s/n Jardim Rosa Elze, S˜ao Crist´ ov˜ao - SE, Brazil 2 Departamento de Matem´atica, Facultad de Ciencias, Universidad del Bio Bio, Casilla 5-C, Concepci´ on, VIII-Regi´ on, Chile Received July 19, 2007; accepted April 14, 2008 Abstract—The problem of knowing the stability of one equilibrium solution of an analytic autonomous Hamiltonian system in a neighborhood of the equilibrium point in the case where all eigenvalues are pure imaginary and the matrix of the linearized system is non-diagonalizable is considered. We give information about the stability of the equilibrium solution of Hamiltonian systems with two degrees of freedom in the critical case. We make a partial generalization of the results to Hamiltonian systems with n degrees of freedom, in particular, this generalization includes those in [1]. MSC2000 numbers: 37C75, 34D20, 34A25 DOI: 10.1134/S1560354708030039 Key words: Hamiltonian system, stability, normal form, resonances 1. INTRODUCTION Consider an autonomous Hamiltonian system with n degrees of freedom ˙ q = ∂H ∂ p (q, p), ˙ p = − ∂H ∂ q (q, p), (1) where H = H (q, p) is a real analytic function of (q, p)=(q 1 , ··· ,q n ,p 1 ··· ,p n ). It is assumed that the origin of the phase space is an equilibrium solution and the Taylor series of H with H (0, 0)=0 in a neighborhood of the origin is H = H (q, p)= H 2 + H 3 + ··· + H j + ··· , (2) where H j represents an homogeneous polynomial of degree j in (q, p), that is, H j =  |k|+|l|=j h kl q k p l , (3) with k =(k 1 , ··· ,k n ) ∈ Z n , l =(l 1 , ··· ,l n ) ∈ Z n , |k| = |k 1 | + ··· + |k n |, |l| = |l 1 | + ··· + |l n |, h kl = h k 1 ···knl 1 ···ln , q k = q k 1 1 ··· q kn n and p l = p l 1 1 ··· p ln n . Assume that the eigenvalues of the linearized system are pure imaginary, say ±ω 1 i,..., ±ω n−2 i, ±ωi, ±ωi, with |ω i | = |ω j | for i = j , i, j ∈{1,...,n − 2}, and that the matrix of the linearized system is non-diagonalizable. In this case we can assume, without loss of generality (see [2] for details), that a linear canonical transformation has already been made such that H 2 = ω 1 2 (q 2 1 + p 2 1 )+ ··· + ω n−2 2 (q 2 n−2 + p 2 n−2 )+ 1 2 (q 2 n−1 + q 2 n )+ ω(q n−1 p n − q n p n−1 ), (4) * E-mail: fsantos@dmat.ufpe.br ** E-mail: clvidal@ubiobio.cl 166