NUMERICAL ANALYSIS OF A FINITE ELEMENT / VOLUME PENALTY METHOD BERTRAND MAURY * Abstract. We present here some contributions to the numerical analysis of the penalty method in the Finite Element context. We are especially interested in the ability provided by this approach to use cartesian, non boundary-fitted meshes, to solve elliptic problems in complicated domain. In the spirit of fictitious domains, the initial problem is replaced by a penalized one, posed over a simply shaped domain which covers the original one. This method relies on two parameters, namely h (space-discretization parameter), and ε (penalty parameter). We propose here a general strategy to estimate the error in both parameters, and we present how it can be applied to various situations. We pay a special attention to a scalar version of the rigid motion constraint for fluid-particle flows. Key words. Finite Element method, penalty, Poisson’s problem, error estimate. AMS subject classifications. 65N30, 65N12, 49M30. 1. Introduction. Because of its conceptual simplicity and the fact that it is straightforward to implement, the penalty method has been widely used to incorporate constraints in numerical optimization. The general principle can been seen as a relaxed version of the following fact: given a proper functional J over a set X , and K a subset of X , minimizing J over K is equivalent to minimizing J K = J + I K over X , where I K is the indicatrix of K: I K (x)=     0 if x ∈ K +∞ if x/ ∈ K Assume now that K is defined as K = {x ∈ X, Ψ(x)=0}, where Ψ is a non-negative function, the penalty method consists in considering relaxed functionals J ε defined as J ε = J + 1 ε Ψ ,ε> 0. By definition of K, the function Ψ/ε approaches I K pointwisely: 1 ε Ψ(x) −→ I K (x) as ε goes to 0 ∀x ∈ X. If J ε admits a minimum u ε , for any ε, one can expect u ε to approach a (or the) minimizer of J over K, if it exists. In the Finite Element context, some u ε h is computed as the solution to a finite di- mensional problem, where h is a space-discretization parameter. The work we present here is motivated by the fact that, even if the penalty method for the continuous prob- lem is convergent and the discretization procedure is sound, the rate of convergence of u ε h toward the exact solution is not straightforward to obtain. A huge litterature is dedicated to the situation where the constraint is distributed over the domain, like the divergence-free constraint for incompressible Stokes flows (see [BF91, GR79]). In this context, the penalty approach makes it possible to use Mixed Finite Element Methods which do not fulfill the so-called Babuska-Brezzi-Ladyzhenskaya (or inf- sup) condition. The penalty approach is also commonly used to prescribe (possibly * Laboratoire de Math´ ematiques, Universit´ e Paris-Sud, 91405 Orsay Cedex. Bertrand.Maury@math.u-psud.fr 1