When agents fail to act: A diagnostic framework for tool invocation reliability in multi-agent LLM systems

Publication Type

Conference Proceeding Article

Publication Date

5-2026

Abstract

Multi-agent systems powered by large language models (LLMs) are transforming enterprise automation, yet systematic evaluation methodologies for assessing tool-use reliability remain underdeveloped. We introduce a comprehensive diagnostic framework that leverages big data analytics to evaluate procedural reliability in intelligent agent systems, addressing critical needs for SME-centric deployment in privacy-sensitive environments. Our approach features a 12-category error taxonomy capturing failure modes across tool initialization, parameter handling, execution, and result interpretation. Through systematic evaluation of 1,980 deterministic test instances spanning both open-weight models (Qwen2.5 series, Functionary) and proprietary alternatives (GPT-4, Claude 3.5/3.7) across diverse edge hardware configurations, we identify actionable reliability thresholds for production deployment. Our analysis reveals that procedural reliability—particularly tool initialization failures—constitutes the primary bottleneck for smaller models, while qwen2.5:32b achieves flawless performance matching GPT-4.1. The framework demonstrates that mid-sized models (qwen2.5:14b) offer practical accuracy-efficiency tradeoffs on commodity hardware (96.6% success rate, 7.3 s latency), enabling cost-effective intelligent agent deployment for resourceconstrained organizations. This work establishes foundational infrastructure for systematic reliability evaluation of toolaugmented multi-agent AI systems.

Keywords

Error Taxonomy, Intelligent Agents, Large Language Models, Multi-Agent Systems, Tool-Use Reliability

Discipline

Artificial Intelligence and Robotics | Software Engineering

Research Areas

Intelligent Systems and Optimization

Publication

Proceedings of the 9th International Conference on Artificial Intelligence and Big Data, ICAIBD 2026, Chengdu, China, May 29-31

First Page

492

Last Page

497

ISBN

9798331582098

Identifier

10.1109/ICAIBD69640.2026.11637248

Publisher

IEEE

City or Country

Pistacataway

Additional URL

https://doi.org/10.1109/ICAIBD69640.2026.11637248

This document is currently not available here.

Share

COinS