EXPLAINHM++: Explainable harmful meme detection with retrieval-augmented debate between large multimodal models

Publication Type

Journal Article

Publication Date

11-2025

Abstract

Identifying harmful memes is challenging due to their implicit meanings, which are not always evident from texts and images alone. Existing solutions often lack clear explanations to justify their decisions. To address this gap, we propose an explainable approach, ExplainHM++, which detects harmful memes by reasoning over competing rationales from both harmful and harmless perspectives. First, inspired by the capabilities of Large Multimodal Models (LMMs) in text generation and multimodal reasoning, we develop ExplainHM, a one-stage multimodal debate in which LMMs generate explanations through contradictory arguments. Second, we fine-tune a small language model to serve as a judge in the debate, improving the integration of harmfulness rationales with the multimodal content of memes. However, we observe that a naive multimodal debate remains vulnerable, as it heavily depends on the inherent reasoning ability of LMMs to understand the memes. Given the evolving and noisy nature of memes, we further introduce a meme sample retrieval mechanism and a retrieval-augmented debate paradigm to strengthen and refine LMM-generated explanations. Extensive experiments on three public meme datasets demonstrate that ExplainHM++ not only outperforms state-of-the-art methods but also provides superior, interpretable explanations for harmful meme detection.

Keywords

Cognition, Visualization, Vaccines, Social Networking Online, Retrieval Augmented Generation, Predictive Models, Electronic Mail

Discipline

Databases and Information Systems

Research Areas

Data Science and Engineering

Publication

IEEE Transactions on Knowledge and Data Engineering

First Page

1

Last Page

14

ISSN

1041-4347

Identifier

10.1109/TKDE.2025.3637552

Publisher

Institute of Electrical and Electronics Engineers

Additional URL

https://doi.org/10.1109/TKDE.2025.3637552

This document is currently not available here.

Share

COinS