Nonlinear Dynamics, Psychology, and Life Sciences, Vol. 17, No. 4, pp. 449-463. © 2013 Society for Chaos Theory in Psychology & Life Sciences. Dynamical Analysis of the Interaction between Effector Immune and Cancer Cells and Optimal Control of Chemotherapy G. H. Erjaee 1 , Shiraz University, Shiraz, Iran, M. H. Ostadzad, Tabriz University, Tabriz, Iran S. Amanpour, Tehran University of Medical Sciences, Tehran, Iran, and K. B. Lankarani, Shiraz University of Medical Sciences, Shiraz, Iran Abstract: It is well known that the tumor chemotherapy treatment has damaging side effects and hence, optimal control of this treatment is extremely important. With this in mind an accurate and comprehensive mathematical model could be useful. Various mathematical models have been derived to describe not only the beneficial effects of the immune system on controlling the growing tumor, but also to track, directly, the detrimental effects of chemotherapy on both the tumor cell and the immune cell populations. In this article, we offer a novel mathematical model presented by fractional differential equations. This model will then be used to analyze the bifurcation and stability of the complex dynamics which occur in the local interaction of effector-immune cell and tumor cells in a solid tumor. We will also investigate the optimal control of combined chemo-immunotherapy. We argue that our fractional differential equations model will be superior to its ordinary differential equations counterpart in facilitating understanding of the natural immune interactions to tumor and of the detrimental side-effects which chemotherapy may have on a patient’s immune system. Key Words: Mathematical Biology, cancer chemotherapy, and optimal drug control. INTRODUCTION Obviously, the primary motivation in producing any mathematical model is to describe a natural or artificial phenomenon by means of a model equation whose behavior is as close as possible to that of the original phenomenon. This is often difficult, however, particularly when dealing with nonlinear behavior in natural complex phenomena such as the interaction between verity of cancers and immune cells. Thus the choice of modeling 1 Correspondence address: G. H. Erjaee, Mathematics Department, Shiraz University, Shiraz, Iran. E-mail: erjaee@shirazu.ac.ir 449