Publication Type

Journal Article

Version

acceptedVersion

Publication Date

11-2026

Abstract

Despite their superb capabilities, Vision-Language Models (VLMs) have been shown to be vulnerable to jailbreak attacks. While recent jailbreaks have achieved notable progress, their effectiveness and efficiency can still be improved. In this work, we reveal an interesting phenomenon: incorporating weak defense cues into the attack pipeline can significantly enhance both the effectiveness and efficiency of jailbreaks on VLMs. Building on this insight, we propose Defense2Attack, a novel jailbreak method that bypasses the safety guardrails of VLMs by leveraging defensive patterns to guide jailbreak prompt construction. Specifically, Defense2Attack consists of three key components: (1) a visual optimizer that embeds universal adversarial perturbations with affirmative and encouraging semantics; (2) a textual optimizer that refines the input using a defense-styled prompt; and (3) a red-team suffix generator that enhances the jailbreak through reinforcement fine-tuning. We empirically evaluate our method on four VLMs and four safety benchmarks. The results demonstrate that Defense2Attack achieves superior jailbreak performance in a single attempt, outperforming state-of-the-art attack methods that often require multiple tries. Our work offers a new perspective on jailbreaking VLMs. Disclaimer: This paper contains content that may be disturbing or offensive.

Keywords

Jailbreak attack, Large Vision-Language Model

Discipline

Graphics and Human Computer Interfaces

Publication

Pattern Recognition

Volume

179

First Page

1

Last Page

9

ISSN

0031-3203

Identifier

10.1016/j.patcog.2026.113805

Publisher

Elsevier

Additional URL

https://doi.org/10.1016/j.patcog.2026.113805

Download

Share

COinS