KYUNGPOOK Math. J. 50(2010), 37-47 The Fekete-Szeg¨ o Problem for a Generalized Subclass of An- alytic Functions Erhan Deniz and Halit Orhan ∗ Department of Mathematics, Faculty of Science, Ataturk University, 25240 Erzu- rum, Turkey e-mail : edeniz36@yahoo.com and horhan@atauni.edu.tr Abstract. In this present work, the authors obtain Fekete-Szeg¨ o inequality for certain normalized analytic function f (z) defined on the open unit disk for which (1-α)z(D m λ,μ f (z)) ′ +αz(D m+1 λ,μ f (z)) ′ (1-α)D m λ,μ f (z)+αD m+1 λ,μ f (z) (λ ≥ μ ≥ 0,m ∈ N0,α ≥ 0) lies in a region starlike with respect to 1 and is symmetric with respect to the real axis. Also certain applications of the main result for a class of functions defined by Hadamard product (or convolution) are given. As a special case of this result, Fekete-Szeg¨ o inequality for a class of func- tions defined through fractional derivatives is obtained. The motivation of this paper is to generalize the Fekete-Szeg¨ o inequalities obtained by Srivastava et al., Orhan et al. and Shanmugam et al., by making use of the generalized differential operator D m λ,μ . 1. Introduction and Definitions Let A denote the class of all analytic functions f (z) of the form: (1.1) f (z)= z + ∞  n=2 a n z n defined on the open unit disk U = {z : z ∈ C and |z| < 1} and let S be the subclass of A consisting of univalent functions. If the functions f (z) and g(z) are analytic in U, we say that f (z) is subordinate to g(z), written symbolically as f ≺ g or f (z) ≺ g(z) ( z ∈ U ) . if there exists a Schwarz function w(z), which (by definition) is analytic in U with w(0) = 0 and |w(z)| < 1 in U such that f (z)= g(w(z)),z ∈ U. For two analytic functions f (z)= z + ∑ ∞ n=2 a n z n and g(z)= z + ∑ ∞ n=2 b n z n , * Corresponding Author. Received November 5, 2008; revised August 12, 2009; accepted October 21, 2009. 2000 Mathematics Subject Classification: 30C45. Key words and phrases: Fekete-Szeg¨ o problem, Analytic functions, Hadamard product, Starlike functions. 37