OPTIMAL CONTROL APPLICATIONS AND METHODS Optim. Control Appl. Meth. (2012) Published online in Wiley Online Library (wileyonlinelibrary.com). DOI: 10.1002/oca.2039 Optimal chemotherapy in cancer treatment: state dependent Riccati equation control and extended Kalman filter Yazdan Batmani * ,† and Hamid Khaloozadeh Department of Systems and Control, Khajeh Nasir Toosi University of Technology, Tehran, Iran SUMMARY In this work, we design a nonlinear state feedback controller based on the State Dependent Riccati Equation (SDRE) technique to eliminate the tumor. One of the most interesting advantages of the SDRE is that it is possible to consider the specific conditions of patients by defining appropriate weights in the cost function and by limiting the administrated drug. Another advantage of this approach is that there are infinite ways to form the state dependent matrices. For each patient, a suitable drug regimen has been obtained using these advantages. A nonlinear model has been utilized to predict the growth of tumor. The model is a system of ODE with four state variables: normal cells, tumor cells, immune cells, and drug concentration. To use the SDRE controller, all state variables must be available for feedback. But for measuring the tumor size, the professional equipment is needed. So, it is impossible to measure the tumor size any time. We suppose that the number of normal cells could be measured in the presence of the Gaussian white noise. Therefore, we can design a state observer to estimate the immeasurable states from measurements. Extended Kalman Filter (EKF) can be used as a state observer for a nonlinear system, and in this work, we use EKF as a nonlinear state observer. Consequently, we can use the SDRE technique just by measuring the normal cell population. Numerical simulations are given to illustrate the design procedure and to show the flexibility of the method. Copyright © 2012 John Wiley & Sons, Ltd. Received 19 March 2011; Revised 14 April 2012; Accepted 25 May 2012 KEY WORDS: adaptive immune system; chemotherapy; SDRE control; extended Kalman filter 1. INTRODUCTION Cancer is one of the most dangerous diseases. Every year in the United States, approximately 1,596,670 people are diagnosed with cancer leading to 571,950 deaths [1]. There are different ways to tackle cancer on the basis of the type of cancer, patient’s conditions, and so on, as surgery, radio- therapy, immunotherapy, chemotherapy, and so on. Chemotherapy is still one of effective methods for cancer treatment. The applied drugs not only destroy the cancerous cells but also do affect the healthy cells and kill them, which are true for all chemotherapy drugs. Therefore, it is important to assure that the patient is able to tolerate the side effects of the drugs. These sorts of side effects cause some restrictions in the dosage of the delivered drugs. The dosage of the prescribed drugs must be as much as the value, which the patient suffers from the minimum side effects, and also, the number of cancerous cells decreases as much as possible. Immune system is a combination of cells, molecules, and different tissues that has the respon- sibility for preventing and eradicating the disease by identifying and killing pathogens and tumor cells. Defensive mechanisms are composed of innate and adaptive immune mechanisms. The innate immune system will act as soon as the infectious agent is emerged. But, this system does not directly *Correspondence to: Yazdan Batmani, Department of Systems and Control, Khajeh Nasir Toosi University of Technol- ogy, Tehran, Iran. † E-mail: yazdan_batmany@ee.kntu.ac.ir Copyright © 2012 John Wiley & Sons, Ltd.