Multi-Agent Reinforcement Learning

The first comprehensive introduction to Multi-Agent Reinforcement Learning (MARL), covering MARL’s models, solution concepts, algorithmic ideas, technical challenges, and modern approaches.

Multi-Agent Reinforcement Learning (MARL), an area of machine learning in which a collective of agents learn to optimally interact in a shared environment, boasts a growing array of applications in modern life, from autonomous driving and multi-robot factories to automated trading and energy network management. This text provides a lucid and rigorous introduction to the models, solution concepts, algorithmic ideas, technical challenges, and modern approaches in MARL. The book first introduces the field’s foundations, including basics of reinforcement learning theory and algorithms, interactive game models, different solution concepts for games, and the algorithmic ideas underpinning MARL research. It then details contemporary MARL algorithms which leverage deep learning techniques, covering ideas such as centralized training with decentralized execution, value decomposition, parameter sharing, and self-play. The book comes with its own MARL codebase written in Python, containing implementations of MARL algorithms that are self-contained and easy to read. Technical content is explained in easy-to-understand language and illustrated with extensive examples, illuminating MARL for newcomers while offering high-level insights for more advanced readers.

• First textbook to introduce the foundations and applications of MARL, written by experts in the field
• Integrates reinforcement learning, deep learning, and game theory
• Practical focus covers considerations for running experiments and describes environments for testing MARL algorithms
• Explains complex concepts in clear and simple language
• Classroom-tested, accessible approach suitable for graduate students and professionals across computer science, artificial intelligence, and robotics
• Resources include code and slides

Table of Contents

1 Introduction

Part 1: Foundations of Multi-Agent Reinforcement Learning

2 Reinforcement Learning

3 Games: Models of Multi-Agent Interaction

4 Solution Concepts for Games

5 Multi-Agent Reinforcement Learning in Games: First Steps and Challenges

6 Multi-Agent Reinforcement Learning: Foundational Algorithms

Part 2: Multi-Agent Deep Reinforcement Learning: Algorithms and Practice

7 Deep Learning

8 Deep Reinforcement Learning

9 Multi-Agent Deep Reinforcement Learning

10 Multi-Agent Deep Reinforcement Learning in Practice

11 Multi-Agent Environments

About the Authors

Stefano V. Albrecht is Associate Professor in the School of Informatics at the University of Edinburgh, where he leads the Autonomous Agents Research Group. His research focuses on the development of machine learning algorithms for autonomous systems control and decision making, with a particular focus on deep reinforcement learning and multi-agent interaction.

Filippos Christianos is a research scientist in multi-agent deep reinforcement learning focusing on how MARL algorithms can be used efficiently and the author of multiple popular MARL-focused code libraries.

Lukas Schäfer is a researcher focusing on the development of more generalizable, robust, and sample-efficient decision making using deep reinforcement learning, with a particular focus on multi-agent reinforcement learning.