Publication Type
Conference Proceeding Article
Version
publishedVersion
Publication Date
4-2025
Abstract
Offline reinforcement learning (RL) has garnered significant attention for its ability to learn effective policies from pre-collected datasets without the need for further environmental interactions. While promising results have been demonstrated in single-agent settings, offline multi-agent reinforcement learning (MARL) presents additional challenges due to the large joint state-action space and the complexity of multi-agent behaviors. A key issue in offline RL is the distributional shift, which arises when the target policy being optimized deviates from the behavior policy that generated the data. This problem is exacerbated in MARL due to the interdependence between agents' local policies and the expansive joint state-action space. Prior approaches have primarily addressed this challenge by incorporating regularization in the space of either Q-functions or policies. In this work, we propose a novel type of regularizer in the space of stationary distributions to address the distributional shift more effectively. Our algorithm, ComaDICE, provides a principled framework for offline cooperative MARL to correct the stationary distribution of the global policy, which is then leveraged to derive local policies for individual agents. Through extensive experiments on the offline multi-agent MuJoCo and StarCraft II benchmarks, we demonstrate that ComaDICE achieves superior performance compared to state-of-the-art offline MARL methods across nearly all tasks.
Keywords
Offline reinforcement learning, Offline multi-agent reinforcement learning, Distributional shift, Stationary distribution regularization, Cooperative MARL
Discipline
Artificial Intelligence and Robotics
Areas of Excellence
Digital transformation
Publication
Proceedings of the 13th International Conference on Learning Representations (ICLR 2025), Singapore, April 24-28
First Page
1
Last Page
15
City or Country
Singapore
Citation
BUI, The Viet; MAI, Tien; and NGUYEN, Hong Thanh.
ComaDICE: Offline cooperative multi-agent reinforcement learning with stationary distribution shift regularization. (2025). Proceedings of the 13th International Conference on Learning Representations (ICLR 2025), Singapore, April 24-28. 1-15.
Available at: https://ink.library.smu.edu.sg/sis_research/10707
Creative Commons License
This work is licensed under a Creative Commons Attribution-NonCommercial-No Derivative Works 4.0 International License.