Applied and Computational Mathematics 2021; 10(3): 56-68 http://www.sciencepublishinggroup.com/j/acm doi: 10.11648/j.acm.20211003.12 ISSN: 2328-5605 (Print); ISSN: 2328-5613 (Online) On Shape Optimization Theory with Fractional Laplacian Malick Fall 1 , Ibrahima Faye 1 , Alassane Sy 1 , Diaraf Seck 2 1 Department of Mathematics, Faculty of Applied Sciences and Information and Communication Technologies, Alioune Diop University, Bambey, Senegal 2 Department of Mathematics of Decision, Faculty of Economics and Management, Cheikh Anta Diop University, Dakar, Senegal Email address: fallm@gmail.com (M. Fall), ibrahima.faye@uadb.edu.sn (I. Faye), alassane.sy@uadb.edu.sn (A. Sy), diaraf.seck@ucad.edu.sn (D. Seck) To cite this article: Malick Fall, Ibrahima Faye, Alassane Sy, Diaraf Seck. On Shape Optimization Theory with Fractional Laplacian. Applied and Computational Mathematics. Vol. 10, No. 3, 2021, pp. 56-68. doi: 10.11648/j.acm.20211003.12 Received: April 17, 2021; Accepted: May 13, 2021; Published: June 26, 2021 Abstract: The fractional Laplacian is a nonlocal operator that appears in biology, in physic, in fluids dynamic, in financial mathematics and probability. This paper deals with shape optimization problem associated to the fractional laplacian Δ s , 0 < s< 1. We focus on functional of the form J (Ω) = j (Ω,u Ω ) where u Ω is solution to the fractional laplacian. A brief review of results related to fractional laplacian and fractional Sobolev spaces are first given. By a variational approach, we show the existence of a weak solution u Ω belonging to the fractional Sobolev spaces D s,2 (Ω) of the boundary value problem considered. Then, we study the existence of an optimal shape of the functional J (Ω) on the class of admissible sets O under constraints volume. Finally, shape derivative of the functional is established by using Hadamard formula’s and an optimality condition is also given. Keywords: Shape Optimisation, Shape Derivative, Optimal Conditions, Fractional Laplacian 1. Introduction In this paper, we are interested for shape optimization problems using fractional laplacian problems. In other words, we look for a domain Ω ⊂ R N ,N ≥ 2 and a function u Ω solutions to the problem inf Ω⊂R N , vol(Ω)=c, ∂Ω∈C 2 J (Ω) = j (Ω,u Ω ) (1) where J (Ω) = C (N, 2) 2  R N  R N |u Ω (x) − u Ω (y)| 2 |x − y| N+2s dxdy (2) and u Ω is solution to  (−Δ) s u Ω = f in Ω u =0 on R N \Ω. (3) where 0 <s< 1, Ω is an open bounded set of R N ,N ≥ 2. Shape optimization problems have always interested the research community. A lot of work related to shape optimization is topical today [14], [2], [4], [5], [7], [6]. Allaire and Henrot [2] give a review on recent development in shape optimization. In general, the functional J depends on Ω and u Ω solution to a partial differential equation. In most of his papers, the authors consider a domain-dependent functionals with constraint a partial differential equation posed in Ω. In general, the solution u Ω of this PDE belongs to a Sobolev space. In this paper, we consider a functional J (Ω) depending on Ω and u Ω solution to the fractional Laplacian. Dalibard and Gerad- Varet in [12], showed that it is possible to calculate the shape derivative of the functional considered in the case s = 1 2 . In this work, we try to generalize the results for all 0 <s< 1. We have the following the main result. Theorem 1.1. Let J (Ω) be a functional given by (2) where u Ω is solution to (3). Then there exists an open set Ω ⊂ R N of class C 2 with vol(Ω) = c satisfying J (Ω) = inf ω⊂R N , vol(ω)=c, ∂ω∈C 2 J (ω) Let’s consider, a small perturbation of the domain Ω in the form Ω t = φ t (Ω) where φ t is a C 1 diffeomorphism such that φ 0 = Id and ∂φt ∂t = V, where V ∈ W 1,∞ (R N , R N ). The shape derivative of the function (2) is given by the following