Self-Organizing Kernel-based Convolutional Echo State Network for Human Actions Recognition Gin Chong Lee 1 , Chu Kiong Loo 2 , Wei Shiung Liew 2 , and Stefan Wermter 3 * 1- Multimedia University - Faculty of Engineering and Technology Jalan Ayer Keroh Lama, 75450 Melaka. - Malaysia 2- University of Malaya - Faculty of Computer Science and Information Technology 50603 Lembah Pantai, Kuala Lumpur. - Malaysia 3- University of Hamburg - Department of Informatics Vogt-Koelln-Str.30, 22527 Hamburg. - Germany Abstract. We propose a deterministic initialization of the Echo State Network reservoirs to ensure that the activation of its internal echo state representations reflects similar topological qualities of the input signal which should lead to a self-organizing reservoir. Human actions encoded as a multivariate time series signal are clustered before using the clus- tered nodes and interconnectivity matrices for initializing the S-ConvESN reservoirs. The capability of S-ConvESN is evaluated using several 3D- skeleton-based action recognition datasets. 1 Introduction Current research in human action recognition (HAR) focuses on the challenge for efficient and effective modeling the temporal features of human actions in 3-dimensional space. Echo state networks (ESNs) are one suitable method for encoding the temporal context due to its short-term memory property. The ran- dom assignment of the ESN’s input and reservoir weights reduces the compu- tational complexity compared to backpropagation but also increases instability and variance in generalization [1]. Using self-organizing kernel networks in the formation of ESN reservoirs ensures that the activation of its internal echo state representations reflects similar topological qualities of the input signal, acting as a feature map which should lead to a self-organizing reservoir [2]. Inspired by the notion that input-dependent self-organization is decisive for the cortex to adjust the neurons according to the distribution of the inputs [3], the potential of unsupervised self-organizing learning seems to be one of the most encouraging and the most biologically plausible. This work proposes an approach to implement a self-organizing kernel net- work in performing deterministic initialization of the input weights and recurrent hidden weights in the ESN stage. This paper is organized as follows: Section 2 briefly discusses the implementation of self-organizing kernel networks, while * This research was supported by the Georg Forster Research Fellowship for Experienced Researchers from Alexander von Humboldt-Stiftung/Foundation, and the ONRG International Fund (IF017-2018). ESANN 2020 proceedings, European Symposium on Artificial Neural Networks, Computational Intelligence and Machine Learning. Online event, 2-4 October 2020, i6doc.com publ., ISBN 978-2-87587-074-2. Available from http://www.i6doc.com/en/. 591