On ξ s -quadratic stochastic operators in 2-dimensional simplex Farrukh Mukhamedov 1 , and Afifah Hanum Mohd Jamal 1 1 International Islamic University Malaysia, 25200 Kuantan, Pahang, Malaysia, farrukh m@iiu.edu.my, mj hanum@yahoo.com Abstract. In this paper we introduce a new class of quadratic stochastic operators called ξ s - QSO. We first classify such operators on 2D-simplex, into six non-isomorphic classes, with respect to their conjugacy and renumeration of the coordinates. Moreover, we investigate the behaviour of operators from two classes. 1 Introduction It is known that there are many systems which are described by nonlinear operators. One of the simplest nonlinear case is quadratic one. Quadratic dynamical systems have been proved to be a rich source of analysis for the investigation of dynamical properties and modeling in different domains. One of such operators is quadratic stochastic operator which naturally arises in modeling of a population dynamics [1]. During many years this theory has developed, and appeared in lots of papers (see e.g. [3–5, 8]). In recent years it has again become of interest in connection with numerous applications to many branches of mathematics, biology and physics. One of the cen- tral problems of this theory is to study the limiting behavior of trajectories of such operators (see [2, 6, 7, 9]). Recall that an evolutionary operator of a free population is a (quadratic) mapping of the simplex S m-1 = {x =(x 1 ,...,x m ) ∈ R m |x i ≥ 0, m  i=1 x i =1} (1) into itself of the form V : x  k = m  i,j =1 P ij,k x i x j , k =1, 2,...,m (2) where P ij,k are coefficient of heredity and P ij,k ≥ 0, P ij,k = P ji,k , m  k=1 P ij,k =1, i, j, k =1, 2,...,m (3) Proceedings of the 6th IMT-GT Conference on Mathematics, Statistics and its Applications (ICMSA2010) Universiti Tunku Abdul Rahman, Kuala Lumpur, Malaysia 159