Copyright © by SIAM. Unauthorized reproduction of this article is prohibited. SIAM J. MATH. ANAL. c  2007 Society for Industrial and Applied Mathematics Vol. 39, No. 3, pp. 687–720 EFFECTIVE TRANSMISSION CONDITIONS FOR REACTION-DIFFUSION PROCESSES IN DOMAINS SEPARATED BY AN INTERFACE ∗ MARIA NEUSS-RADU † AND WILLI J ¨ AGER ‡ Abstract. In this paper, we develop multiscale methods appropriate for the homogenization of processes in domains containing thin heterogeneous layers. Our model problem consists of a nonlinear reaction-diffusion system defined in such a domain, and properly scaled in the layer region. Both the period of the heterogeneities and the thickness of the layer are of order ε. By performing an asymptotic analysis with respect to the scale parameter ε we derive an effective model which consists of the reaction-diffusion equations on two domains separated by an interface together with appropriate transmission conditions across this interface. These conditions are determined by solving local problems on the standard periodicity cell in the layer. Our asymptotic analysis is based on weak and strong two-scale convergence results for sequences of functions defined on thin heterogeneous layers. For the derivation of the transmission conditions, we develop a new method based on test functions of boundary layer type. Key words. nonlinear reaction-diffusion systems, thin heterogeneous layer, homogenization, two-scale convergence, transmission conditions AMS subject classifications. 35K57, 35B27, 80M35 DOI. 10.1137/060665452 1. Introduction. In this paper, we will be concerned with a nonlinear system of reaction-diffusion equations in a domain containing a thin heterogeneous layer. Such problems often occur in applications like, e.g., transdermal diffusion of drugs, diffusion of substances through the epithelial monolayer, and transport of ions through membranes. We start from a microscopic model defined on a domain containing a thin layer of thickness ε. The processes are modeled by a system of reaction-diffusion equations properly scaled inside the layer. Our aim is to study the behavior of the solutions of the microscopic equations when the thickness ε tends to zero. In the limit ε → 0 the thin layer reduces to an interface between the two bulk regions. We derive an effective model which consists of a system of reaction-diffusion equations on both sides of this interface together with appropriate transmission con- ditions for the limit concentrations across the interface. For the derivation of the limit equations in the bulk regions we use standard com- pactness results based on classical a priori estimates. However, the thin heterogeneous layer poses additional problems. We have to adapt the concepts of weak and strong two-scale convergences to functions on thin domains with oscillatory and (for sim- plicity) periodic structure. Due to the arising nonlinearities, it is necessary to prove strong two-scale convergence of the solutions to the ε-problems in the thin layer. To this end, we introduce macroscopic and microscopic coordinates and analyze the reg- ularity of the solutions with respect to this pair of coordinates. Whereas the gradients ∗ Received by the editors July 19, 2006; accepted for publication (in revised from) March 12, 2007; published electronically August 17, 2007. http://www.siam.org/journals/sima/39-3/66545.html † IAM, University of Heidelberg, INF 294, 69120 Heidelberg, Germany (maria.neuss-radu@iwr.uni- heidelberg.de). ‡ IWR, University of Heidelberg, INF 368, 69120 Heidelberg, Germany (jaeger@iwr.uni-heidelberg. de). 687