Noname manuscript No. (will be inserted by the editor) Learning the Coordinate Gradients Yiming Ying † · Qiang Wu ‡ · Colin Campbell † Received: January 2009 / Accepted: date Abstract In this paper we study the problem of learning the gradient function with application to variable selection and determining variable covariation. Firstly, we pro- pose a novel unifying framework for coordinate gradient learning from the perspective of multi-task learning. Various variable selection algorithms can be regarded as spe- cial instances of this framework. Secondly, we formulate the dual problems of gradient learning with general loss functions. This enables the direct application of standard optimization toolboxes to the case of gradient learning. For instance, gradient learning with SVM loss can be solved by quadratic programming (QP) routines. Thirdly, we propose a novel gradient learning algorithm which can be cast as learning the kernel matrix problem. Its relation with sparse regularization is highlighted. A semi-infinite linear programming (SILP) approach and an iterative optimization approach are pro- posed to efficiently solve this problem. Finally, we validate our proposed approaches on both synthetic and real datasets. Keywords Learning the Gradient · Multi-task Kernel · Feature Selection · Sparse Regularization · Learning the Kernel Matrix Mathematics Subject Classification (2000) 68T05 · 68Q32 · 68T10 · 62P10 1 Introduction Kernel methods [30,31], such as Support Vector Machines (SVMs), have been success- fully demonstrated in many supervised learning tasks. In this case, we are interested in learning an appropriate function for regression or classification. However, in many applications we are not only interested in learning a target function, but also wish to learn salient coordinate variables and their interaction with each other. This problem † Department of Engineering Mathematics, University of Bristol, Queen’s Building, Bristol, BS8 1TR, UK Tel.: +44(0)117 331 7379 Fax: +44 (0)117 925 1154 E-mail: {enxyy,C.Campbell}@bris.ac.uk · ‡ Departments of Mathematics, Michigan State University, East Lansing, MI 48824, USA E-mail: wuqiang@math.msu.edu