355 0022-4715/03/0400-0355/0 © 2003 Plenum Publishing Corporation Journal of Statistical Physics, Vol. 111, Nos. 1/2, April 2003 (© 2003) Analyticity of Smooth Eigenfunctions and Spectral Analysis of the Gauss Map I. Antoniou 1–3 and S. A. Shkarin 1, 2 1 International Solvay Institutes for Physics and Chemistry, Campus Plaine ULB C.P.231, Bd.du Triomphe, Brussels 1050, Belgium; e-mail: iantonio@vub.ac.be 2 Department of Mathematics and Mechanics, Moscow State University, Vorobjovy Gory, Moscow 119899, Russia; e-mail: shkarin@math.uni-wuppertal.de, sshkarin@hotmail.com 3 Department of Mathematics, Aristotle University of Thessaloniki, 54006, Greece. Received July 18, 2001; accepted October 10, 2002 We provide a sufficient condition of analyticity of infinitely differentiable eigenfunctions of operators of the form Uf(x)=> a(x, y) f(b(x, y)) m(dy) acting on functions f: [u, v] Q C (evolution operators of one-dimensional dynamical systems and Markov processes have this form). We estimate from below the region of analyticity of the eigenfunctions and apply these results for studying the spectral properties of the Frobenius–Perron operator of the continuous fraction Gauss map. We prove that any infinitely differentiable eigenfunction f of this Frobenius–Perron operator, corresponding to a non-zero eigenvalue admits a (unique) analytic extension to the set C 0 (-., -1]. Analyzing the spectrum of the Frobenius–Perron operator in spaces of smooth functions, we extend significantly the domain of validity of the Mayer and Röpstorff asymptotic formula for the decay of correlations of the Gauss map. KEY WORDS: Gauss map; Frobenius–Perron operators; analytic extension; decay of correlations; spectral decomposition. 1. INTRODUCTION The Gauss or continuous fractions map G: (0, 1) Q [0, 1), G(x)=1/x (mod 1) (1) is one of the most interesting exact dynamical systems with origin not only in number theory (1–4) but also in cosmology since G is an approximation of the Poincare return map of the Mixmaster cosmological model. For the