Copyright © by SIAM. Unauthorized reproduction of this article is prohibited. SIAM J. OPTIM. c  2009 Society for Industrial and Applied Mathematics Vol. 20, No. 1, pp. 416–444 SHAPE OPTIMIZATION IN THREE-DIMENSIONAL CONTACT PROBLEMS WITH COULOMB FRICTION ∗ P. BEREMLIJSKI † , J. HASLINGER ‡ , M. KO ˇ CVARA § , R. KU ˇ CERA ¶ , AND J.V. OUTRATA ‖ Abstract. We study the discretized problem of the shape optimization of three-dimensional (3D) elastic bodies in unilateral contact. The aim is to extend existing results to the case of contact problems obeying the Coulomb friction law. Mathematical modeling of the Coulomb friction problem leads to an implicit variational inequality. It is shown that for small coefficients of friction the discretized problem with Coulomb friction has a unique solution and that this solution is Lipschitzian as a function of a control variable describing the shape of the elastic body. The 2D case of this problem was studied by the authors in [P. Beremlijski, J. Haslinger, M. Koˇ cvara, and J. V. Outrata, SIAM J. Optim., 13 (2002), pp. 561–587]; there we used the so-called implicit programming approach combined with the generalized differential calculus of Clarke. The extension of this technique to the 3D situation is by no means straightforward. The main source of difficulties is the nonpolyhedral character of the second-order (Lorentz) cone, arising in the 3D model. To facilitate the computation of the subgradient information, needed in the used numerical method, we exploit the substantially richer generalized differential calculus of Mordukhovich. Numerical examples illustrate the efficiency and reliability of the suggested approach. Key words. shape optimization, contact problems, Coulomb friction, mathematical programs with equilibrium constraints AMS subject classifications. 49Q10, 74M10, 74S05 DOI. 10.1137/080714427 1. Introduction and preliminaries. Contact shape optimization is a special branch of structural optimization whose goal is to find shapes of deformable bodies which are in mutual contact. A typical problem in many applications is to find shapes guaranteeing a priori given stress distributions on parts in contact [1]. A specific feature of contact shape optimization is its nonsmooth character due to the fact that the respective state mapping is given by various types of variational inequalities. For contact problems without friction or with the so-called given friction (see [9]), whose mathematical models lead to variational inequalities of the first and the second kind, sensitivity analysis was done in [26] for continuous models and in [10] for discretized models. Assuming a more realistic Coulomb law of friction, the situation becomes much more complicated in view of the fact that the state problem is now represented ∗ Received by the editors January 27, 2008; accepted for publication (in revised form) December 8, 2008; published electronically April 29, 2009. This work was supported by grants IAA1075402 and IAA100750802 of the Czech Academy of Sciences (JH, MK, JVO), grant 201/07/0294 of the Grant Agency of the Czech Republic (PB), research projects MSM6198910027 (PB, RK) and MSM0021620839 (JH) of the Czech Ministry of Education, and the EU Commission in the Sixth Framework Program, project 30717 PLATO-N (MK). http://www.siam.org/journals/siopt/20-1/71442.html † Faculty of Electrical Engineering, V ˇ SB-Technical University of Ostrava, 17. listopadu 15, 70833 Ostrava-Poruba, Czech Republic (petr.beremlijski@vsb.cz). ‡ Department of NumericalMathematics, Charles University, Sokolovsk´a 83, 18675 Praha 8, Czech Republic (Jaroslav.Haslinger@mff.cuni.cz). § School of Mathematics, University of Birmingham, Birmingham B15 2TT, UK and Institute of Information Theory and Automation, Academy of Sciences of the Czech Republic, Pod vod´arenskou vˇ eˇ z´ ı 4, 182 08 Praha 8, Czech Republic (kocvara@maths.bham.ac.uk). ¶ Department of Mathematics and Descriptive Geometry, V ˇ SB-Technical University of Ostrava, 17. listopadu 15, 708 33 Ostrava-Poruba, Czech Republic (radek.kucera@vsb.cz). ‖ Institute of Information Theory and Automation, Academy of Sciences of the Czech Republic, Pod vod´arenskou vˇ eˇ z´ ı 4, 182 08 Praha 8, Czech Republic (outrata@utia.cas.cz). 416