PREPRINT SUBMITTED TO IEEE TRANSACTIONS ON COGNITIVE AND DEVELOPMENTAL SYSTEMS 1 Incremental procedural and sensorimotor learning in cognitive humanoid robots Leonardo de Lellis Rossi, Let´ ıcia Mara Berto, Eric Rohmer, Paula Paro Costa, Ricardo Ribeiro Gudwin, Esther Luna Colombini and Alexandre da Silva Sim˜ oes Abstract—The ability to automatically learn movements and behaviors of increasing complexity is a long-term goal in au- tonomous systems. Indeed, this is a very complex problem that involves understanding how knowledge is acquired and reused by humans as well as proposing mechanisms that allow artificial agents to reuse previous knowledge. Inspired by Jean Piaget’s theory’s first three sensorimotor substages, this work presents a cognitive agent based on CONAIM (Conscious Attention- Based Integrated Model) that can learn procedures incremen- tally. Throughout the paper, we show the cognitive functions required in each substage and how adding new functions helps address tasks previously unsolved by the agent. Experiments were conducted with a humanoid robot in a simulated environment modeled with the Cognitive Systems Toolkit (CST) performing an object tracking task. The system is modeled using a single procedural learning mechanism based on Reinforcement Learn- ing. The increasing agent’s cognitive complexity is managed by adding new terms to the reward function for each learning phase. Results show that this approach is capable of solving complex tasks incrementally. Index Terms—Cognitive Robotics, Cognitive Architectures, Reinforcement Learning, Incremental Learning, Developmental Robotics. I. I NTRODUCTION A DVANCEMENTS in artificial intelligence and robotics increased the interest in introducing robots into daily activities that involve interaction with other agents, both robots and humans. These robots should operate autonomously in complex, partially unknown, unpredictable, and unstructured scenarios, making pre-programming impossible and requiring robots to have a superior capability to perform tasks. This challenge raises questions such as how to incorporate new knowledge and skills through interactions with the world, resulting in the research area of Cognitive Robotics. Cognitive Robotics is intrinsically related to Cognitive Architectures (CA), which represent comprehensive computer models pro- viding theoretical frameworks to work with cognitive pro- cesses searching for complex behavior. This work was developed within the scope of PPI-Softex with support from MCTI through the Technical Cooperation Term [01245.013778/2020-21]. A. S. Sim˜ oes and L. L. Rossi are with Dept. of Control and Automation Engineering (DECA), Institute of Science and Technology (ICT), Campus Sorocaba, Universidade Estadual Paulista (Unesp) R. R. Gudwin, E. Rohmer, P. P. Costa and L. L. Rossi are with the Dept. of Computer Engineering and Industrial Automation (DCA), School of Electrical and Computer Engineering (FEEC), University of Campinas, Brazil E. L. Colombini and L. M. Berto are with the Laboratory of Robotics and Cognitive Systems (LaRoCS), Institute of Computing, University of Campinas, Brazil All authors are with the Artificial Intelligence and Cognitive Architectures Hub (H.IAAC), University of Campinas, Brazil Cognitive architectures are systems that can reason in differ- ent domains, develop different views, adapt to new situations, and reflect on themselves [1], [2]. They are general control systems inspired by scientific theories developed to explain cognition in humans and other animals, comprising modules responsible for implementing different cognitive abilities, such as perception, attention, memory, reasoning, and learning. Inspired by how humans build knowledge through interac- tions with the world, cognitive architecture researchers seek to reproduce this behavior with artificial creatures [3]. However, the development of cognitive skills in machines requires the coordination of complex mechanisms that depend on each other. According to Piaget [4], the process of developing these skills is incremental and evolutionary. In this work, a cognitive agent based on the CONAIM model (Conscious Attention-Based Integrated Model)[2] was proposed and implemented with the Cognitive Systems Toolkit [3]. A humanoid robot was designed to incrementally learn procedures to perform object tracking experiments inspired by the first three sensorimotor substages of Jean Piaget’s Theory [4]. Throughout the work, we present the cognitive functions necessary to form circular reactions in each substage using a Reinforcement Learning (RL) [5] environment and how new functions can be added to the reward function allowing the agent to solve complex tasks, previously unresolved. As the main contributions of this work, we can list the following: 1) The proposition of a cognitive architecture based on CONAIM with attention, memories, and learning mod- ules focused on sensorimotor and procedural learning; 2) The design and implementation of CONAIM’s top-down pathway in CST that can be incorporated into any agent implemented with CST; 3) The design and implementation of a single procedural learning mechanism in CST that can incrementally learn and reuse schemas for the first three sensorimotor sub- stages of Piaget’s Theory; 4) The modeling of a set of environments for sensorimotor experiments for the movements learning in humanoid robots; 5) The implementation and evaluation of sensorimotor ex- periments for object tracking in the first three sensori- motor substages of Piaget’s Theory as proposed by [6]. The code used to implement the architecture is available at: https://github.com/CST-Group/cst. arXiv:2305.00597v1 [cs.RO] 30 Apr 2023