ORIGINAL PAPER Modulation stability analysis and solitary wave solutions of nonlinear higher-order Schro ¨dinger dynamical equation with second-order spatiotemporal dispersion A R Seadawy 1,2 * , M Arshad 3 * and D Lu 3 * 1 Mathematics Department, Faculty of Science, Taibah University, Medina, Saudi Arabia 2 Mathematics Department, Faculty of Science, Beni-Suef University, Beni Suef, Egypt 3 Faculty of Science, Jiangsu University, Zhenjiang 212013, Jiangsu, People’s Republic of China Received: 15 April 2018 / Accepted: 18 September 2018 Abstract: In optical fibers, the higher-order nonlinear Schro ¨dinger (NLS) dynamical equation which describes the beyond the classic slowly varying envelopes and spatiotemporal dispersion of pulses is investigated. By applying the proposed modified extended mapping method, the optical soliton solutions of higher-order NLS dynamical equation with the coefficients of group velocity dispersion, second-order spatiotemporal dispersion and cubic nonlinearity are deduced. The obtained solutions have important applications in applied sciences and engineering. The formation conditions are specified on parameters in which optical solitons can exist for this media. The moments of some constructed solutions are presented graphically which facilitate the researchers to comprehend the physical phenomena of this equation. The modulation instability analysis is utilized to discuss the model stability, which verifies that all obtained solutions are stable and exact. Other such forms of the system arising in sciences and engineering can also be solved by this steadfast, influential and effective method. Keywords: Modified extended mapping method; Higher-order nonlinear Schro ¨dinger equation; Solitons; Solitary wave solutions PACS Nos.: 02.30.Jr; 05.45.Yv; 47.10.A?; 47.35.?i; 47.35.Fg 1. Introduction The nonlinear Schro ¨dinger equations (NLSEs) having the coefficients of group velocity dispersion and second-order spatiotemporal dispersion are important physical models and illustrate the dynamics of optical soliton promulgation in the optical fibers for trans-continental communication [1–5]. In optical fibers, most of these models are regularly expressed in the time domain, and when fields at different frequencies propagate through the fiber the common practice is also to write a distance equation for each field component. The nonlinear transformation of dielectric of the fiber termed as the Kerr effect is applied to neutralize the dispersion effect; in this state, the optical pulse might lean to form a steady nonlinear pulse known as an optical soliton. The bit rate of transmission is restricted by the dispersion of the fiber material. The fiber loss is the only factor that contributes to the drop in the pulse quality by expansion in the pulse width (for more details see refer- ences [6–8]). In optical fibers, the dynamical models of soliton propagation are an area of enormous curiosity because of the broad applications in ultrafast signal routing systems, trans-continental and short-light-pulse telecommunication [9, 10]. These systems are mostly articulated in time domain, and when different frequency fields propagate throughout the fiber the ordinary practice is also to inscribe a distance equation for every field component. In dielectric fibers, the authors in [10] examined the optical solitons experimentally and theoretically. Solitons in homogeneous and Hamiltonian systems are localized solitary waves *Corresponding author, E-mail: Aly742001@yahoo.com; muham- mad.arshad18@yahoo.com; dclu@ujs.edu.cn Indian J Phys https://doi.org/10.1007/s12648-018-01361-y Ó 2019 IACS