PHYSICAL REVIEW E 84, 061131 (2011) Non-Markovian models for migration-proliferation dichotomy of cancer cells: Anomalous switching and spreading rate Sergei Fedotov, 1 Alexander Iomin, 2 and Lev Ryashko 3 1 School of Mathematics, The University of Manchester, Manchester M60 1QD, United Kingdom 2 Department of Physics, Technion, Haifa 32000, Israel 3 Department of Mathematical Physics, Ural Federal University, Ekaterinburg, Russia (Received 3 August 2011; revised manuscript received 1 November 2011; published 19 December 2011) Proliferation and migration dichotomy of the tumor cell invasion is examined within two non-Markovian models. We consider the tumor spheroid, which consists of the tumor core with a high density of cells and the outer invasive zone. We distinguish two different regions of the outer invasive zone and develop models for both zones. In model I we analyze the near-core-outer region, where biased migration away from the tumor spheroid core takes place. We suggest non-Markovian switching between the migrating and proliferating phenotypes of tumor cells. Nonlinear master equations for mean densities of cancer cells of both phenotypes are derived. In anomalous switching case we estimate the average size of the near-core-outer region that corresponds to sublinear growth 〈r (t )〉∼ t μ for 0 <μ< 1. In model II we consider the outer zone, where the density of cancer cells is very low. We suggest an integrodifferential equation for the total density of cancer cells. For proliferation rate we use the classical logistic growth, while the migration of cells is subdiffusive. The exact formulas for the overall spreading rate of cancer cells are obtained by a hyperbolic scaling and Hamilton-Jacobi techniques. DOI: 10.1103/PhysRevE.84.061131 PACS number(s): 05.40.Fb, 87.15.Vv, 87.17.Ee I. INTRODUCTION Clinical investigations of a glioma cancer show that the proliferation rate of migratory cells is essentially lower in the invasion region than in the tumor core [1,2]. This phenomenon is known as the migration-proliferation dichotomy. Proliferation and migration of cells are mutually exclusive: the high motility suppresses cell proliferation and vice versa. This finding triggered extensive theoretical studies that resulted in several phenomenological models [3–14]. A switching process between two phenotypes still is not well understood, and a lot of efforts are taken to develop relevant models with different mechanisms of switching of the glioma cells. It was suggested by Khain et al. [3,4] that the motility of cancer cells is a function of their density. Multiparametric modeling of the phenotype switching was considered in Ref. [5]. The agent-based approach to simulate multiscale glioma growth and invasion was used in Refs. [6,7]. Subdiffusive cancer development on a comb was studied in Ref. [8]. A stochastic approach for the proliferation-migration switching involving only two parameters was proposed in Refs. [9,10] where the transport of cancer cells was formulated in terms of a continuous time random walk (CTRW). A “go or grow” mechanism was proposed in Ref. [11], where the transition to invasive tumor phenotypes can be explained on the basis of the oxygen shortage in the environment of a growing tumor. Phenotypic switching due to density effect was also suggested in Refs. [12,13]. Both numerical and analytical approaches were developed in Ref. [14] to study the glioma propagation in the framework of reaction-diffusion equations, where the phenotype switching depends on oxygen in a threshold manner. Collective behavior of brain tumor cells under the hypoxia condition was studied in Ref. [15]. We should also mention the cellular automaton modeling for tumor invasion [16]. The multiscale approaches for modeling of tumor growth was reviewed in Ref. [17]. One of the main features of malignant brain cancer is the ability of tumor cells to invade the normal tissue away from the multicell tumor core, and the motility is the most critical feature of brain cancer, causing treatment failure [18]. The main problem in glioma treatment is how to distinguish the genuine boundaries of the invaded area. There is a need for a proper description of cancer cell motility. As shown in Refs. [19,20], and then verified in Refs. [9,10], the standard diffusion approximation for the transport together with a logistic growth yields an overestimation of the overall propagation rate. The main reason for employing the CTRW models [21–23], beyond the standard diffusion approximation, is to give the mesoscopic description of cell motility by taking into account memory effects [24] and anomalous dynamics of cell migration [25–27]. To describe a migration-proliferation dichotomy, one can use the standard phenomenological model involving reaction- diffusion equations. In this model one assumes that the cancer cells can be in two states: mobile state (migratory phenotype) and immobile state (proliferating phenotype). If we introduce the density of the cells of migrating phenotype, n 1 (t,x), and the density of the cells of proliferating phenotype, n 2 (t,x), then the system of equations can be written as ∂n 1 ∂t + ∇ · (vn 1 ) = ∇ · (D∇n 1 ) − β 1 n 1 + β 2 n 2 , (1) ∂n 2 ∂t = f (n)n 2 + β 1 n 1 − β 2 n 2 , (2) where v is the advective velocity, D is the diffusion coefficient. The switching between two phenotypes is determined by the switching rates β 1 and β 2 . The nonlinear function f (n) is the proliferation rate, where n = n 1 + n 2 . For example, the logistic growth rate corresponds to f (n) = U  1 − n K  , (3) 061131-1 1539-3755/2011/84(6)/061131(8) ©2011 American Physical Society