SIAM J. CONTROL OPTIM. c  2014 Society for Industrial and Applied Mathematics Vol. 52, No. 5, pp. 3371–3400 SHAPE OPTIMIZATION IN CONTACT PROBLEMS WITH COULOMB FRICTION AND A SOLUTION-DEPENDENT FRICTION COEFFICIENT ∗ P. BEREMLIJSKI † , J. HASLINGER ‡ , J. V. OUTRATA § , AND R. PATH ´ O ¶ Abstract. The present paper deals with shape optimization in discretized two-dimensional (2D) contact problems with Coulomb friction, where the coefficient of friction is assumed to de- pend on the unknown solution. Discretization of the continuous state problem leads to a system of finite-dimensional implicit variational inequalities, parametrized by the so-called design variable, that determines the shape of the underlying domain. It is shown that if the coefficient of friction is Lipschitz and sufficiently small in the C 0,1 -norm, then the discrete state problems are uniquely solvable for all admissible values of the design variable (the admissible set is assumed to be com- pact), and the state variables are Lipschitzian functions of the design variable. This facilitates the numerical solution of the discretized shape optimization problem by the so-called implicit program- ming approach. Our main results concern sensitivity analysis, which is based on the well-developed generalized differential calculus of B. Mordukhovich and generalizes some of the results obtained in this context so far. The derived subgradient information is then combined with the bundle trust method to compute several model examples, demonstrating the applicability and efficiency of the presented approach. Key words. shape optimization, contact problems, Coulomb friction, solution-dependent coef- ficient of friction, mathematical programs with equilibrium constraints AMS subject classifications. 49Q10, 74M10, 74S05 DOI. 10.1137/130948070 1. Introduction. Contact shape optimization is a branch of optimal control theory in which the control variables, called in this context the design variables, are linked to the geometry of elastic bodies that are in mutual contact. By changing their shapes only, one strives to achieve the best possible or some a priori given properties of the system. A design is evaluated by the so-called cost functional that is subject to minimization. Common examples include minimizing the normal stress along the contact surface (related to the minimization of the potential energy) or attaining a given contact stress profile; see, e.g., [1]. Physical quantities, subject to ∗ Received by the editors December 6, 2013; accepted for publication (in revised form) July 11, 2014; published electronically October 28, 2014. The second, third, and fourth authors were sup- ported by grants P201/12/0671 of the Grant Agency of the Czech Republic. http://www.siam.org/journals/sicon/52-5/94807.html † Centre of Excellence, IT4Innovations and Department of Applied Mathematics, V ˇ SB- Technical University of Ostrava, 17. listopadu 15/2172, 708 33 Ostrava-Poruba, Czech Republic (petr.beremlijski@vsb.cz). This author was supported by the European Regional Development Fund in the Centre of Excellence project IT4Innovations (CZ.1.05/1.1.00/02.0070) and by the project SPOMECH - Creating a multidisciplinary R&D team for reliable solution of mechanical problems, reg. no. CZ.1.07/2.3.00/20.0070 within Operational Programme “Education for competitiveness” funded by Structural Funds of the European Union and state budget of the Czech Republic. ‡ Department of Numerical Mathematics, Charles University in Prague, Sokolovsk´a 83, 186 75 Praha 8, Czech Republic (hasling@karlin.mff.cuni.cz). § Institute of Theory of Information and Automation of the AS CR, Pod Vod´ arenskou vˇ eˇ z´ ı 4, 182 08 Praha 8, Czech Republic, and Graduate School of Information Technology and Mathematical Sciences, University of Ballarat, Ballarat VIC 3350, Australia (outrata@utia.cas.cz). This author was supported by the ARC project DP110102011. ¶ Department of Numerical Mathematics, Charles University in Prague, Sokolovsk´a 83, 186 75 Praha 8, Czech Republic (patho@karlin.mff.cuni.cz). This author was supported by GAUK no. 719912. 3371