RESEARCH PAPER Enhancing Emotion Detection with Non-invasive Multi-Channel EEG and Hybrid Deep Learning Architecture Durgesh Nandini 1 • Jyoti Yadav 1 • Asha Rani 1 • Vijander Singh 1 Received: 19 September 2023 / Accepted: 19 February 2024 Ó The Author(s), under exclusive licence to Shiraz University 2024 Abstract Emotion recognition is vital for augmenting human–computer interactions by integrating emotional contextual information for enhanced communication. Hence, the study presents an intelligent emotion detection system developed utilizing hybrid stacked gated recurrent units (GRU)-recurrent neural network (RNN) deep learning architecture. Integration of GRU with RNN allows the system to make use of both models’ capabilities, making it better at capturing complex emotional patterns and temporal correlations. The EEG signals are investigated in time, frequency, and time–frequency domains, meticulously curated to capture intricate multi-domain patterns. Then, the SMOTE-Tomek method ensures a uniform class distribution, while the PCA technique optimizes features by minimizing data redundancy. A comprehensive experimentation including the well-established emotion datasets: DEAP and AMIGOS, assesses the efficacy of the hybrid stacked GRU and RNN architecture in contrast to 1D convolution neural network, RNN and GRU models. Moreover, the ‘‘Hyperopt’’ technique fine-tunes the model’s hyperparameter, improving the average accuracy by about 3.73%. Hence, results revealed that the hybrid GRU-RNN model demonstrates the most optimal performance with the highest classification accuracies of 99.77% ± 0.13, 99.54% ± 0.16, 99.82% ± 0.14, and 99.68% ± 0.13 for the 3D VAD and liking parameter, respectively. Furthermore, the model’s generalizability is examined using the cross-subject and database analysis on the DEAP and AMIGOS datasets, exhibiting a classification with an average accuracy of about 99.75% ± 0.10 and 99.97% ± 0.03. Obtained results when compared with the existing methods in literature demonstrate superior performance, highlighting potential in emotion recognition. Keywords DEAP EEG database  Emotion detection  HCI  Affective computing  GRU-RNN  3D VAD model  Hyperopt technique 1 Introduction Emotions are a foundational component of human exis- tence, playing a pivotal role in an individual’s inherent reactions to a spectrum of life’s occurrences. They signif- icantly influence one’s daily decision-making processes and, consequently, shape overall life experiences. The recent advancements in artificial intelligence (AI) have enabled several Human–Computer-Interactions (HCI) sys- tems, evoking the need for emotionally aware interactions between humans and machines. Affective computing, an integral AI domain, analyses, interprets and discerns human emotions. It comprehends human emotions by identifying emotional cues that emerge during human– computer contact and synthesizes an emotional response to develop emotionally intelligent machines. Humanoid & Durgesh Nandini durgesh.ic18@nsut.ac.in Jyoti Yadav bmjyoti@gmail.com Asha Rani asha.rani@nsut.ac.in Vijander Singh vijaydee@nsut.ac.in 1 Department of Instrumentation and Control Engineering, Netaji Subhas University of Technology, Sector 3, Dwarka, New Delhi, India 123 Iranian Journal of Science and Technology, Transactions of Electrical Engineering https://doi.org/10.1007/s40998-024-00710-4