Publication Type
Conference Proceeding Article
Version
publishedVersion
Publication Date
9-2023
Abstract
Learning control policies for a large number of agents in a decentralized setting is challenging due to partial observability, uncertainty in the environment, and scalability challenges. While several scalable multiagent RL (MARL) methods have been proposed, relatively few approaches exist for large scale constrained MARL settings. To address this, we first formulate the constrained MARL problem in a collective multiagent setting where interactions among agents are governed by the aggregate count and types of agents, and do not depend on agents’ specific identities. Second, we show that standard Lagrangian relaxation methods, which are popular for single agent RL, do not perform well in constrained MARL settings due to the problem of credit assignment—how to identify and modify behavior of agents that contribute most to constraint violations (and also optimize primary objective alongside)? We develop a fictitious MARL method that addresses this key challenge. Finally, we evaluate our approach on two large-scale real-world applications: maritime traffic management and vehicular network routing. Empirical results show that our approach is highly scalable, can optimize the cumulative global reward and effectively minimize constraint violations, while also being significantly more sample efficient than previous best methods.
Keywords
Constraint optimization, Multi-agent systems, Multiagent reinforcement learning
Discipline
Artificial Intelligence and Robotics | Operations Research, Systems Engineering and Industrial Engineering
Research Areas
Intelligent Systems and Optimization
Publication
Machine Learning and Knowledge Discovery in Databases: European Conference, ECML PKDD 2022, Grenoble, France, September, 19-23: Proceedings
Volume
13716
First Page
183
Last Page
199
ISBN
9783031264115
Identifier
10.1007/978-3-031-26412-2_12
Publisher
Springer
City or Country
Cham
Citation
LING, Jiajing; SINGH, Arambam James; NGUYEN, Duc Thien; and KUMAR, Akshat.
Constrained multiagent reinforcement learning for large agent population. (2023). Machine Learning and Knowledge Discovery in Databases: European Conference, ECML PKDD 2022, Grenoble, France, September, 19-23: Proceedings. 13716, 183-199.
Available at: https://ink.library.smu.edu.sg/sis_research/8091
Creative Commons License
This work is licensed under a Creative Commons Attribution-NonCommercial-No Derivative Works 4.0 International License.
Additional URL
https://doi.org/10.1007/978-3-031-26412-2_12
Included in
Artificial Intelligence and Robotics Commons, Operations Research, Systems Engineering and Industrial Engineering Commons