SIAM J. CONTROL OPTIM. c  2009 Society for Industrial and Applied Mathematics Vol. 48, No. 4, pp. 2562–2580 IMPULSE RESPONSE APPROXIMATIONS OF DISCRETE SHAPE HESSIANS WITH APPLICATION IN CFD ∗ STEPHAN SCHMIDT † AND VOLKER SCHULZ † Abstract. This paper discusses the symbol of the Hessian of a shape optimization problem in a viscid, incompressible flow. The symbol of the Hessian for the Stokes shape optimization problem is analytically approximated by tracking a Fourier mode through the operators involved. We propose a discrete method for finding the symbol and confirm the correctness by comparison with the analytical data. A preconditioner is constructed for both the Stokes and Navier–Stokes equation, which greatly accelerates the optimization. Key words. shape Hessian, operator symbol, preconditioning, Fourier mode, shape optimiza- tion, approximative Newton method, Stokes, Navier–Stokes AMS subject classifications. 65K10, 93C80, 35S99, 49M25, 76D55, 49Q10 DOI. 10.1137/080719844 1. Introduction. The majority of shape optimization problems in CFD are currently conducted by means of parameterized gradients. The shape of an obstacle in a flow is discretized by a certain parameterization, mainly b-splines or Hicks–Henne functions in 2D and freeform deformation in 3D. The gradients are computed with respect to these design parameters. This has several distinct disadvantages: First, the choice of parameterization limits the search space; e.g., it is next to impossible to compute an optimal shape with a sharp nose if the b-splines used cannot generate such a shape. Next, the computation of these gradients is usually not independent of the number of design parameters, because certain so-called “mesh sensitivities” must be computed, which is usually done via finite differences or automatic differentiation techniques. Lastly, the parameterized gradients camouflage the shape Hessian of the problem. The remedy for these disadvantages is the use of shape derivatives, as they have been proposed for aerodynamic optimization in [1, 2, 3, 4, 8]. The goal of this paper is to both analytically and discretely analyze the shape Hessian of a drag minimization problem in a viscous and incompressible flow and to use the information gathered to construct a preconditioner which greatly accelerates the optimization procedure by turning a steepest descent algorithm into an approximative Newton scheme. This procedure is often called “gradient smoothing,” and the smoothed gradients are also often referred to as “Sobolev gradients.” Usually, the gradients are smoothed by us- ing a scaled tangential Laplace or Laplace–Beltrami operator as a preconditioning PDE [10, 11, 12] or some other ODE [9]. Other approaches use multilevel informa- tion [5]. However, the proper choice of the scaling or “smoothing” parameter and the PDE to use should be done by taking the Hessian operator of the problem into account. As such, the main focus of this paper is the shape Hessian and its relation to the preconditioning PDE. Interestingly, the problems discussed here will provide an example where the tangential Laplacian—which is mostly used—will result in an ∗ Received by the editors March 31, 2008; accepted for publication (in revised form) May 21, 2009; published electronically September 4, 2009. http://www.siam.org/journals/sicon/48-4/71984.html † Department of Mathematics, University of Trier, 54286 Trier, Germany (Stephan.Schmidt@uni- trier.de, Volker.Schulz@uni-trier.de). 2562