Received: 6 July 2017 Revised: 22 August 2017 Accepted: 24 August 2017 DOI: 10.1002/num.22205 RESEARCH ARTICLE Dynamical study of two predators and one prey system with fractional Fourier transform method Kolade M. Owolabi 1 Edson Pindza 2 Matt Davison 2 1 Institute for Groundwater Studies, Faculty of Natural and Agricultural Sciences, University of the Free State, Bloemfontein 9300, South Africa 2 Department of Statistical and Actuarial Sciences, University of Western Ontario, London, ON, Canada N6A 5B7 Correspondence Edson Pindza, Department of Statistical and Actuarial Sciences, University of Western Ontario, London, ON, Canada N6A 5B7. Email: pindzaedson@yahoo.fr In this work, we investigate both the analytical and numer- ical studies of the dynamical model comprising of three species systems. We analyze the linear stability of stationary solutions in the one-dimensional multisystem modeling the interactions of two predators and one prey species. The sta- bility analysis has a lot of implications for understanding the various spatiotemporal and chaotic behaviors of the species in the spatial domain. The analysis results presented have established the possibility of the three-interacting species to coexist harmoniously, this feat is achieved by combining the local and global analyses to determine the global dynamics of the system. In the presence of a fractional diffusion term, we introduced a fractional Fourier transform for solving the system modeled by fractional partial differential equations. The main advantages of this method are that it yields a fully diagonal representation of the fractional operator with expo- nential accuracy and a completely straightforward extension to high spatial dimensions. The scheme is described in detail and justified by a number of computational experiments. KEYWORDS coexistence, exponential time differencing method, fractional Fourier transform, global and local stability, nonlinear, predator-prey model, reaction-diffusion system 1 INTRODUCTION Evolution equations containing fractional derivatives are becoming increasingly used as a power- ful modeling approach for understanding the many aspects of nonlocality and spatial heterogeneity. Numer Methods Partial Differential Eq. 2017;1–23. wileyonlinelibrary.com/journal/num © 2017 Wiley Periodicals, Inc. 1