DISCRETE AND CONTINUOUS doi:10.3934/dcdsb.2018196 DYNAMICAL SYSTEMS SERIES B Volume 24, Number 2, February 2019 pp. 547–561 COSYMMETRY APPROACH AND MATHEMATICAL MODELING OF SPECIES COEXISTENCE IN A HETEROGENEOUS HABITAT Alexander V. Budyansky Don State Technical University, Rostov-on-Don, 344002, Russia Kurt Frischmuth University of Rostock D-18051 Rostock, Germany Vyacheslav G. Tsybulin * Southern Federal University Rostov-on-Don, 344090, Russia (Communicated by Chris Cosner) Abstract. We explore an approach based on the theory of cosymmetry to model interaction of predators and prey in a two-dimensional habitat. The model under consideration is formulated as a system of nonlinear parabolic equations with spatial heterogeneity of resources and species. Firstly, we ana- lytically determine system parameters, for which the problem has a nontrivial cosymmetry. To this end, we formulate cosymmetry relations. Next, we em- ploy numerical computations to reveal that under said cosymmetry relations, a one-parameter family of steady states is formed, which may be characterized by different proportions of predators and prey. The numerical analysis is based on the finite difference method (FDM) and staggered grids. It allows to follow the transformation of spatial patterns with time. Eventually, the destruction of the continuous family of equilibria due to mistuned parameters is analyzed. To this end, we derive the so-called cosymmetric selective equation. Investigation of the selective equation gives an insight into scenarios of local competition and coexistence of species, together with their connection to the cosymmetry relations. When the cosymmetry relation is only slightly violated, an effect we call ’memory on the lost family’ may be observed. Indeed, in this case, a slow evolution takes place in the vicinity of the lost states of equilibrium. 1. Introduction. The preservation of biodiversity requires a scientific understand- ing of the processes driving the changes of spatial distributions of species, e.g. preda- tors and prey, both inhabiting a common biotope. In this context, the application of systems of parabolic partial differential equations describing the dynamics of pop- ulations is widely used [6, 22]. Most approaches to modeling the spatial migration of species are based on homogeneous diffusion, [1, 12, 20, 24, 28]. However, from 2010 Mathematics Subject Classification. Primary: 35Q92, 92D25; Secondary: 37M05. Key words and phrases. population kinetics, prey-predator model, nonlinear parabolic equa- tions, cosymmetry, finite difference method, multistability. AVB and VGT are supported by Russian Foundation for Basic Research (grant 18-01-00453). * Corresponding author. 547