A New Architecture for Optimization Modeling Frameworks Matt Wytock, Steven Diamond, Felix Heide, Stephen Boyd Department of Electrical Engineering Stanford University {mwytock, diamond, fheide, boyd}@stanford.edu Abstract—We propose a new architecture for optimiza- tion modeling frameworks in which solvers are expressed as computation graphs in a framework like TensorFlow rather than as standalone programs built on a low-level linear algebra interface. Our new architecture makes it easy for modeling frameworks to support high performance com- putational platforms like GPUs and distributed clusters, as well as to generate solvers specialized to individual prob- lems. Our approach is particularly well adapted to first- order and indirect optimization algorithms. We introduce cvxflow, an open-source convex optimization modeling framework in Python based on the ideas in this paper, and show that it outperforms the state of the art. I. I NTRODUCTION Optimization offers a principled approach to solving problems in a wide variety of application domains, such as machine learning, statistics, control, signal and image processing, networking, engineering design, finance, and many others [BV04]. Instead of designing specialized al- gorithms for each individual problem, the user describes the problem as the minimization of a cost function and the optimal solution with minimal cost is found by the optimization method. The wealth of applications for this methodology has driven the development of several high-level modeling languages. These languages provide a separation of con- cerns between the development of mathematical models and the implementation of numerical routines to opti- mize these models. This is especially useful for rapidly prototyping new applications, allowing practitioners to easily experiment with different cost functions and con- straints by writing expressions that closely mimic the mathematical optimization problem. Prominent examples of modeling languages and frameworks include AMPL [FGK02], YALMIP [Lof04], and CVX [GB14], as well as several tied closely to particular solvers, such as CPLEX’s ILOG [ILO07] and MathProg from GLPK [Mak00]. Despite the popularity of these modeling frameworks, support for modern large-scale computational environ- ments such as GPUs and distributed clusters is virtually nonexistent. In part, this is due to fundamental challenges in scaling interior point methods, which have historically been the basis for solvers of most modeling frameworks, as these methods require solving sparse linear systems to high accuracy and as such do not benefit greatly from GPU implementation. In addition, distributing such methods beyond a single machine typically requires high bandwidth interconnects such as those available exclusively in HPC environments. However, there are also highly practical reasons for the lack of support for new environments: mature solvers of- ten require several years to develop and writing entirely new implementations of low-level numerical routines specialized to each environment is unappealing. Tradi- tionally, a degree of platform independence has been provided by implementing on top of low-level linear al- gebra libraries (e.g., BLAS, LAPACK, and SuiteSparse), but as we discuss in this paper, this architecture is often insufficient, especially for large problems. In addition, such libraries do not handle memory management and data transfer between GPU and CPU or between multiple machines. The solution that we explore is a new architecture for optimization modeling frameworks based on solvers represented as computation graphs. This architecture is well-suited for solving large optimization problems by taking advantage of problem-specific structure. In particular, the computation graph abstraction naturally represents the composition of structured linear operators which can be significantly more efficient than the stan- dard sparse or dense matrix representation. We develop such a method in this paper and demonstrate that it outperforms the existing state of the art for solving large convex optimization problems, a GPU-enabled version of SCS [OCPB16], which itself is one of the few GPU- PyHPC2016; Salt Lake City, Utah, USA; November 2016 978-1-5090-5220-2/16/$31.00 c 2016 IEEE arXiv:1609.03488v2 [math.OC] 11 Oct 2016