Soft Computing https://doi.org/10.1007/s00500-018-3615-x METHODOLOGIES AND APPLICATION Optimization of support vector machine parameters for voltage stability margin assessment in the deregulated power system G. S. Naganathan 1 · C. K. Babulal 2 © Springer-Verlag GmbH Germany, part of Springer Nature 2018 Abstract Recently, the electric power systems are operated relatively close to their operational limits due to worldwide deregulated electricity market policies. The power systems are being operated with high stress, and hence sufficient voltage stability margin is necessary to be managed to ensure secure operation of the power system. A particle swarm optimization-based support vector machine (SVM) approach for online monitoring of voltage stability has been proposed in this paper. The conventional methods for voltage stability monitoring are less accurate and highly time-consuming consequently, infeasible for online application. SVM is a powerful machine learning technique and widely used in power system to predict the voltage stability margin, but its performances depend on the selection of parameters greatly. So, the particle swarm optimization is applied to determine the parameter settings of SVM. The proposed approach uses bus voltage angle and reactive power load as the input vectors to SVM, and the output vector is the voltage stability margin index. The effectiveness of the proposed approach is tested using the IEEE 14-bus test system, IEEE 30-bus test system and the IEEE 118-bus test system. The results of the proposed PSO-SVM approach for voltage stability monitoring are compared with artificial neural networks and grid search SVM approach with same data set to prove its superiority. Keywords Particle swarm optimization (PSO) · Support vector machine (SVM) · Voltage stability margin index (VSMI) · Artificial neural networks (ANN) · Grid search (GS) 1 Introduction The electric power industry has changed significantly due to the restructuring of regulated power sector to separate the functions of power generation, transmission, distribution and electricity supply to consumers (Miranda 2003). Because of less regulation in power flow patterns and more intensive use of available transmission facilities through bilateral and multilateral transactions in deregulated power systems, the Communicated by V. Loia. B G. S. Naganathan naganathangs@gmail.com C. K. Babulal ckbeee@tce.edu 1 Department of Electrical and Electronics Engineering, Syed Ammal Engineering College, Ramanathapuram, Tamilnadu 623502, India 2 Department of Electrical and Electronics Engineering, Thiagarajar College of Engineering, Madurai, Tamilnadu 625015, India systems are operated closer to the voltage stability bound- aries (Chung et al. 2004). Voltage stability refers to the ability of a power system to maintain acceptable voltages at all buses both under normal operating conditions and after being subject to a disturbances (VanCutsem and Vournas 1998). A power system enters a state of voltage instabil- ity when disturbing it, which results in a progressive and an uncontrollable voltage decline leading to voltage collapse (Taylor 1994; Kundur 1994). Voltage collapse is a major cause for many power system blackouts around the world (Taylor 1994). In order to prevent the occurrence of voltage collapse, it is essential to accurately predict the operating condition of a power system. So, independent system opera- tors (ISO) need a fast and accurate voltage stability index to help them for monitoring the system condition. Many authors have proposed the voltage stability indices based on repeated power flow analysis (Moghavvemi and Omar 1998; Musirin and Rahman 2002; Bansilal et al. 2003). The main difficulty in these methods is that Jacobian matrix of power flow equation becomes singular at the voltage stability limit. Singularity in the Jacobian matrix can be avoided by slightly 123