Robotics and Autonomous Systems 35 (2001) 109–122 Modular Q-learning based multi-agent cooperation for robot soccer Kui-Hong Park, Yong-Jae Kim, Jong-Hwan Kim ∗ Department of Electrical Engineering and Computer Science, Korea Advanced Institute of Science and Technology (KAIST), Kusong-dong, Yusong-gu, Taejon-shi 305-701, South Korea Received 8 August 2000; received in revised form 12 February 2001 Communicated by F.C.A. Groen Abstract In a multi-agent system, action selection is important for the cooperation and coordination among agents. As the environment is dynamic and complex, modular Q-learning, which is one of the reinforcement learning schemes, is employed in assigning a proper action to an agent in the multi-agent system. The architecture of modular Q-learning consists of learning modules and a mediator module. The mediator module of the modular Q-learning system selects a proper action for the agent based on the Q-value obtained from each learning module. To obtain better performance, along with the Q-value, the mediator module also considers the state information in the action selection process. A uni-vector field is used for robot navigation. In the robot soccer environment, the effectiveness and applicability of modular Q-learning and the uni-vector field method are verified by real experiments using five micro-robots. © 2001 Elsevier Science B.V. All rights reserved. Keywords: Multi-agent system; Robot soccer system; Reinforcement learning; Modular Q-learning; Action selection 1. Introduction It is important that multi-agent systems perform tasks that are complex and difficult. This needs co- operation and coordination among the agents [3,9]. Developing a multi-agent system amounts to the search for a method for implementing an intelligent system composed of multi-agents, with independent motion control and cooperation with each other. Multi-agent systems are more flexible and fault tol- erant as several simple robot agents are easier to handle and cheaper to build compared to a single ∗ Corresponding author. E-mail addresses: khpark@vivaldi.kaist.ac.kr, (K.-H. Park), johkim@vivaldi.kaist.ac.kr (J.-H. Kim). powerful robot which can carry out different tasks [7]. From the standpoint of multi-agent systems, robot soccer is a good example of the problems in real world which can be moderately modeled. The soccer game is different from other multi-agent systems in that the robots of one team have to cooperate while facing competition with the opponent team. The co- operative and competitive strategies used play a major role in a robot soccer system [10]. The related re- search issues are quite wide and they are associated with the hardware configuration, software implemen- tation, agent/robot communication, sensor fusion and learning, to mention a few. The action of the robot is usually selected by considering some conditions in the robot soccer 0921-8890/01/$ – see front matter © 2001 Elsevier Science B.V. All rights reserved. PII:S0921-8890(01)00114-2