Privacy-Preservation for Gradient Descent Methods ∗ Li Wan School of Computer Engineering Nanyang Technological University Singapore wanl0001@ntu.edu.sg Wee Keong Ng School of Computer Engineering Nanyang Technological University Singapore awkng@ntu.edu.sg Shuguo Han School of Computer Engineering Nanyang Technological University Singapore hans0004@ntu.edu.sg Vincent C. S. Lee School of Business Systems Monash University Australia vincent.lee@infotech.monash.edu.au ABSTRACT Gradient descent is a widely used paradigm for solving many optimization problems. Stochastic gradient descent per- forms a series of iterations to minimize a target function in order to reach a local minimum. In machine learning or data mining, this function corresponds to a decision model that is to be discovered. The gradient descent paradigm un- derlies many commonly used techniques in data mining and machine learning, such as neural networks, Bayesian net- works, genetic algorithms, and simulated annealing. To the best of our knowledge, there has not been any work that extends the notion of privacy preservation or secure multi- party computation to gradient-descent-based techniques. In this paper, we propose a preliminary approach to enable privacy preservation in gradient descent methods in general and demonstrate its feasibility in specific gradient descent methods. Categories and Subject Descriptors H.2.8 [Database Management]: Database Applications— Data mining ; H.2.7 [Database Management]: Database Administration—Security, integrity, protection General Terms Theory, Algorithms, Security Keywords Privacy Preservation, Gradient Descent Method, Secure Multi- party Computation, Regression ∗ This work is supported in part by grant P0520095 from the Agency for Science, Technology and Research (A*STAR), Singapore. Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. To copy otherwise, to republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. KDD'07, August 12–15, 2007, San Jose, California, USA. Copyright 2007 ACM 978-1-59593-609-7/07/0008 ...$5.00. 1. INTRODUCTION Many techniques in data mining and machine learning follow a gradient descent paradigm in the iterative process of discovering a target function or decision model. For in- stance, neural networks generally perform a series of iter- ations to converge the weight coefficients of edges in the network; thus, settling into a decision model. Linear re- gression is a basic statistical method that finds a function to correlate two or more attributes. Linear regression can also be resolved through a gradient descent method that it- eratively minimizes the error of the target function. Other gradient-descent-based methods include Bayesian networks induction, genetic algorithms, and simulated annealing. Secure multi-party computation and privacy preservation have attracted much attention recently in incorporating se- curity into data mining and machine learning algorithms. A key issue in multi-party secure methods is to allow in- dividual parties to preserve the privacy of its data, while contributing to the computation of a global result together with other parties. Many methods have been proposed to perform Secure Multi-party Computation (SMC) on various basic operations required in data mining. For instance, the scalar product is a basic operation in inducing decision trees and association rule mining that can now be performed se- curely involving two or more parties [1, 3]. Basic matrix op- erations such as matrix multiplication and matrix inversion have also been extended in a secure manner for preserving privacy in various statistical methods [2]. To the best of our knowledge, there has not been any work that extends privacy preservation or secure multi-party com- putation to gradient descent methods. Our contributions in this paper are as follows: 1. We propose a generic formulation of gradient descent methods for secure computation by defining the target function f as a composition g ◦h(a 1 ,a 2 ,...,a m ), where g is any differentiable function and h = P m j=1 h j (a j ,ω j ) is linearly separable. 2. With this formulation, we propose a secure two-party protocol for performing gradient descent. We show that the protocol is correct and privacy preserving. We then extend the protocol to perform secure multi- party gradient descent. We demonstrate how the generic secure gradient descent Research Track Paper 775