Brain tumor classification from MRI scans: a framework of hybrid deep learning model with Bayesian optimization and quantum theory-based marine predator algorithm Muhammad Sami Ullah 1 , Muhammad Attique Khan 1 , Anum Masood 2 * , Olfa Mzoughi 3 , Oumaima Saidani 4 and Nazik Alturki 4 1 Department of Computer Science, HITEC University, Taxila, Pakistan, 2 Department of Physics, Norwegian University of Science and Technology, Trondheim, Norway, 3 Department of Computer Science, College of Computer Engineering and Sciences, Prince Sattam bin Abdulaziz University, Al -Kharj, Saudi Arabia, 4 Department of Information Systems, College of Computer and Information Sciences, Princess Nourah Bint Abdulrahman University, Riyadh, Saudi Arabia Brain tumor classification is one of the most difficult tasks for clinical diagnosis and treatment in medical image analysis. Any errors that occur throughout the brain tumor diagnosis process may result in a shorter human life span. Nevertheless, most currently used techniques ignore certain features that have particular significance and relevance to the classification problem in favor of extracting and choosing deep significance features. One important area of research is the deep learning-based categorization of brain tumors using brain magnetic resonance imaging (MRI). This paper proposes an automated deep learning model and an optimal information fusion framework for classifying brain tumor from MRI images. The dataset used in this work was imbalanced, a key challenge for training selected networks. This imbalance in the training dataset impacts the performance of deep learning models because it causes the classifier performance to become biased in favor of the majority class. We designed a sparse autoencoder network to generate new images that resolve the problem of imbalance. After that, two pretrained neural networks were modified and the hyperparameters were initialized using Bayesian optimization, which was later utilized for the training process. After that, deep features were extracted from the global average pooling layer. The extracted features contain few irrelevant information; therefore, we proposed an improved Quantum Theory-based Marine Predator Optimization algorithm (QTbMPA). The proposed QTbMPA selects both networks’ best features and finally fuses using a serial-based approach. The fused feature set is passed to neural network classifiers for the Frontiers in Oncology frontiersin.org 01 OPEN ACCESS EDITED BY Poonam Yadav, Northwestern University, United States REVIEWED BY Summrina Kanwal, Independent Researcher, Helsinborg, Sweden L. J. Muhammad, Bayero University Kano, Nigeria *CORRESPONDENCE Anum Masood anum.masood@ntnu.no RECEIVED 09 November 2023 ACCEPTED 12 January 2024 PUBLISHED 08 February 2024 CITATION Ullah MS, Khan MA, Masood A, Mzoughi O, Saidani O and Alturki N (2024) Brain tumor classification from MRI scans: a framework of hybrid deep learning model with Bayesian optimization and quantum theory-based marine predator algorithm. Front. Oncol. 14:1335740. doi: 10.3389/fonc.2024.1335740 COPYRIGHT © 2024 Ullah, Khan, Masood, Mzoughi, Saidani and Alturki. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) and the copyright owner(s) are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms. TYPE Original Research PUBLISHED 08 February 2024 DOI 10.3389/fonc.2024.1335740