TRANSACTIONS OF THE AMERICAN MATHEMATICAL SOCIETY Volume 303, Number 1, September 1987 ON THE GENERALIZED SPECTRUM FOR SECOND-ORDER ELLIPTIC SYSTEMS ROBERT STEPHEN CANTRELL AND CHRIS COSNER ABSTRACT. We consider the system of homogeneous Dirichlet boundary value problems (*) Liu = A[an(x)u + a\2(x)v], L2V = ti[ai2{x)u + a.22{x)v] in a smooth bounded domain fi C R", where Li and £2 are formally self- adjoint second-order strongly uniformly elliptic operators. Using linear per- turbation theory, continuation methods, and the Courant-Hilbert variational eigenvalue characterization, we give a detailed qualitative and quantitative de- scription of the real generalized spectrum of (*), i.e., the set {(A, fi) € R2 : (*) has a nontrivial solution}. The generalized spectrum, a term introduced by Protter in 1979, is of considerable interest in the theory of linear partial dif- ferential equations and also in bifurcation theory, as it is the set of potential bifurcation points for associated semilinear systems. 1. Introduction. Suppose that fi is a bounded smooth domain in Rw, N > 1, and that Lt, i = 1,2, are second-order strongly uniformly elliptic operators acting on functions from fi into C. Consider then the system (1.1) Lyu = \[an(x)u + ai2(x)v], L2v = fi[a2i(x)u + 022(2;)?;] in fi, where u and v are required to satisfy homogeneous Dirichlet boundary con- ditions. A point (A,/i) G C2 for which (1.1) has a nontrivial solution is called a point of the generalized spectrum for (1.1). This term was introduced in 1979 by Protter [14] for a class of problems which includes (1.1). He found that "the process for obtaining lower bounds for the spectrum of a second order system is improved substantially by the introduction of [this] generalization of the spectrum." From a different though related point of view, generalized spectra represent the potential primary bifurcation points to associated semilinear problems. Such prob- lems provide good examples for the recently developed multiparameter bifurcation theory (see, for example, Alexander and Antman [1, 2], Fitzpatrick, Massabo, and Pejsachowicz [10, 11], and Ize, Massabo, Pejsachowicz, and Vignoli [13]). Further, such semilinear systems determine the steady-states to reaction-diffusion systems arising in the applications. In particular, the situation when L\ — L2 — —A and diffusion coefficients are allowed to vary independently from equation to equation occurs frequently. For instance, Brown and Eilbeck [3] exploit the generalized spectrum to study stabiltity properties of constant solutions to the problem (1.2) ut(x,t)=diAu{x,t) +F(u,v), vt{x, t) = d2Av(x, t) + G(u, v) Received by the editors May 1, 1986 and, in revised form, October 6, 1986. The contents of this paper were presented by the second author at the meeting on Ordinary Differential Equations, March 22-28, 1987, sponsored by the Mathematisches Forschungsinstitut Oberwolfach. 1980 Mathematics Subject Classification (1985 Revision). Primary 35P99, 35J55. ©1987 American Mathematical Society 0002-9947/87 $1.00 + $.25 per page 345 License or copyright restrictions may apply to redistribution; see http://www.ams.org/journal-terms-of-use