Publication Type

Conference Proceeding Article

Version

publishedVersion

Publication Date

9-2024

Abstract

Multi-agent systems (MASs) have achieved remarkable success in multi-robot control, intelligent transportation, and multiplayer games, etc. Thorough testing for MAS is urgently needed to ensure its robustness in the face of constantly changing and unexpected scenarios. Existing methods mainly focus on single-agent system testing and cannot be directly applied to MAS testing due to the complexity of MAS. To our best knowledge, there are fewer studies on MAS testing. While several studies have focused on adversarial attacks on MASs, they primarily target failure detection from an attack perspective, i.e., discovering failure scenarios, while ignoring the diversity of scenarios. In this paper, to highlight a typical balance between exploration (diversifying behaviors) and exploitation (detecting failures), we propose an advanced testing framework for MAS called with diversity-guided exploration and adaptive critical state exploitation. It incorporates both individual diversity and team diversity, and designs an adaptive perturbation mechanism to perturb the action at the critical states, so as to trigger more and more diverse failure scenarios of the system. We evaluate MASTest on two popular MAS simulation environments: Coop Navi and StarCraft II. Results show that the average distance of the resulting failure scenarios is increased by 29.55%-103.57% and 74.07%-370.00% on two environments compared to the baselines. Also, the failure patterns found by MASTest are improved by 71.44%-300.00% and 50%-500.00% on two experimental environments compared to the baselines.

Keywords

Adaptive perturbation exploitation, Critical-state, Diversity-guided exploration, Failure scenarios, Intelligent transportation, Multi-agent system testing, Multiagent systems (MASs), Multirobots, Robots control, System testing

Discipline

Robotics | Software Engineering

Research Areas

Data Science and Engineering; Information Systems and Management

Publication

Proceedings of the 33rd ACM SIGSOFT International Symposium on Software Testing and Analysis, Vienna, Austria, 2024 September 16-20

First Page

1491

Last Page

1503

ISBN

9798400706127

Identifier

10.1145/3650212.3680376

Publisher

ACM

City or Country

New York

Additional URL

https://doi.org/10.1145/3650212.3680376

Download

Share

COinS