Genetic Structural NAS: A Neural Network Architecture Search with Flexible Slot Connections Jakub Sadel KP Labs Gliwice, Poland jsadel@kplabs.pl Michal Kawulok Silesian University of Technology Gliwice, Poland michal.kawulok@polsl.pl Mateusz Przeliorz KP Labs Gliwice, Poland mprzeliorz@kplabs.pl Jakub Nalepa Silesian University of Technology Gliwice, Poland jakub.nalepa@polsl.pl Daniel Kostrzewa ∗ Silesian University of Technology Gliwice, Poland daniel.kostrzewa@polsl.pl ABSTRACT Selecting an appropriate neural network architecture and hyper- parameters to optimize performance for a given application is a difcult task. To overcome this challenge, Neural Architecture Search (NAS) and Hyperparameter Optimization (HPO) have been introduced. However, these techniques are often computationally- expensive and require signifcant amounts of execution time. To address this issue, we propose a semi-automated NAS approach that optimizes pre-existing architectural structures using genetic algo- rithms and eliminates unsuccessful combinations through shallow network training. The efectiveness of our technique was verifed through an experiment that produces a family of AlexNet-like neu- ral networks comprising 1296 models for image classifcation tasks. The computational study was conducted with fve runs of a genetic algorithm, resulting in the deep models with mean loss of 0.7044 and accuracy of 0.7334, both with low standard deviations, outper- forming the original AlexNet model with a signifcant margin. CCS CONCEPTS · Computing methodologies → Neural networks; Genetic algorithms;· Computer systems organization → Neural net- works. KEYWORDS neural networks, genetic algorithm, neural architecture search ACM Reference Format: Jakub Sadel, Michal Kawulok, Mateusz Przeliorz, Jakub Nalepa, and Daniel Kostrzewa. 2023. Genetic Structural NAS: A Neural Network Architecture Search with Flexible Slot Connections. In Genetic and Evolutionary Com- putation Conference Companion (GECCO ’23 Companion), July 15ś19, 2023, Lisbon, Portugal. ACM, New York, NY, USA, 2 pages. https://doi.org/10.1145/ 3583133.3596435 ∗ Corresponding author. Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for proft or commercial advantage and that copies bear this notice and the full citation on the frst page. Copyrights for third-party components of this work must be honored. For all other uses, contact the owner/author(s). GECCO ’23 Companion, July 15ś19, 2023, Lisbon, Portugal © 2023 Copyright held by the owner/author(s). ACM ISBN 979-8-4007-0120-7/23/07. https://doi.org/10.1145/3583133.3596435 1 INTRODUCTION Deep neural networks have been widely recognized for their re- markable success in various felds, particularly in computer vision and natural language processing, as they have become state-of- the-art solutions for tackling complex tasks. The ability of neural networks to learn intricate features and patterns from large datasets has led to signifcant breakthroughs in many domains. Despite their practical utility, selecting an appropriate network architecture and associated hyperparameters that can optimize their performance for a given task remains an open challenge. This is where Neural Architecture Search (NAS) [4] and Hyperparameter Optimization (HPO) [6] come into play by providing automated techniques for identifying the best-performing network architec- ture and hyperparameters for a task at hand. NAS automates the design process of a network’s architecture, while HPO fne-tunes the hyperparameters of a given deep network architecture to boost its performance. While these operations are currently being carried out in an automated manner in the majority of cases, due to large search spaces, they require signifcant amounts of time and com- putational power. To overcome this challenge, various techniques are employed, including shallow training, exploiting pre-trained modules, as well as utilizing (meta)heuristics for faster convergence. In this paper, we introduce a semi-automated NAS approach that can enhance the performance of networks deep within a short time frame. Specifcally, our method optimizes pre-existing structures using genetic algorithms and eliminates unsuccessful combinations through shallow network training, providing a practical and ef- cient solution for boosting the deep network’s performance. 2 THE METHOD The approach proposed in this work is based on creating a main network structure consisting of multiple slots that can be fexibly connected to one another. A similar partitioning can already be observed in the feld of object detection, where feature extraction and classifcation components were identifed as part of the model- building process. However, in our solution, there is a great deal of fexibility aforded in the slot connections. In contrast to the approach introduced in [5], a single slot in this methodology can, but does not necessarily have to be, a complex structure. Instead, it can equally be a single layer within the network. In the context of optimizing a (deep) neural network, it is im- portant to consider three main components: the main network 79