Publication Type

Conference Proceeding Article

Version

publishedVersion

Publication Date

4-2026

Abstract

Penetration testing is essential for identifying vulnerabilities in web applications before real adversaries can exploit them. Recent work has explored automating this process with Large Language Model (LLM)-powered agents, but existing approaches either rely on a single generic agent that struggles in complex scenarios or narrowly specialized agents that cannot adapt to diverse vulnerability types. We therefore introduce PenForge, a framework that dynamically constructs expert agents during testing rather than relying on those prepared beforehand. By integrating automated reconnaissance of potential attack surfaces with agents instantiated on the fly for context-aware exploitation, PenForge achieves a 30.0% exploit success rate (12/40) on CVE-Bench in the particularly challenging zero-day setting, which is a 3 × improvement over the state-of-the-art. Our analysis also identifies three opportunities for future work: (1) supplying richer tool-usage knowledge to improve exploitation effectiveness; (2) extending benchmarks to include more vulnerabilities and attack types; and (3) fostering developer trust by incorporating explainable mechanisms and human review. As an emerging result with substantial potential impact, PenForge embodies the early-stage yet paradigm-shifting idea of on-the-fly agent construction, marking its promise as a step toward scalable and effective LLM-driven penetration testing.

Discipline

Artificial Intelligence and Robotics | Information Security | Software Engineering

Research Areas

Intelligent Systems and Optimization

Areas of Excellence

Digital transformation

Publication

ICSE-NIER '26: Proceedings of the IEEE/ACM 48th International Conference on Software Engineering, Rio de Janeiro, Brazil, April 12-18

First Page

76

Last Page

80

ISBN

9798400724251

Identifier

10.1145/3786582.3786814

Publisher

ACM

City or Country

New York

Additional URL

https://doi.org/10.1145/3786582.3786814

Download

Share

COinS