Cross-modal explicit invariance coordinated dual constraint reconstruction for heterogeneous incomplete multimodal sentiment analysis

Publication Type

Journal Article

Publication Date

1-2026

Abstract

Multimodal data holds great potential in enhancing the accuracy of sentiment analysis. However, the multimodal sentiment analysis (MSA) task faces two practical challenges: i) multimodal heterogeneity significantly increases the difficulty of integrating multimodal information, resulting in insufficient multimodal learning; and ii) multimodal incompleteness easily triggers cross-modal inference biases, seriously hindering multimodal understanding. In this paper, we propose a novel multimodal framework named c ross-modal explicit in varianc ecoordinated dual constraint reconstruction (Clive-Dcr) to mitigate MSA for heterogeneous incomplete multimodal data. Specifically, we present a language-oriented cross-modal invariance reasoning (LCIR) approach that can alleviate the inter-modal representation gaps under the supervision of fine-grained sentiment information. More importantly, LCIR can explicitly quantify inter-modal consistency and break the vagueness of heterogeneous modal information, thereby fully exploring the complementary advantages between invariant features and specific features. In the prediction head, we develop a multimodal purification learning mechanism to minimize the intervention of sentiment-irrelevant information on MSA. To enhance the robustness under incomplete modalities, we design a dual constraint reconstruction (DCR) strategy. During the reconstruction of missing features, DCR not only performs high-level feature distribution alignment but also extends the cross-modal invariance reasoning from the complete to the incomplete multimodal scenarios. Comprehensive experiments on three multimodal datasets verify that our Clive-Dcr achieves remarkable performance under complete and incomplete multimodal patterns.

Keywords

Cross-modal invariance reasoning, Dual constraint reconstruction, Heterogeneous incomplete multimodal data, Multimodal purification learning, Multimodal sentiment analysis

Discipline

Artificial Intelligence and Robotics

Research Areas

Data Science and Engineering

Publication

IEEE Transactions on Multimedia

Volume

28

First Page

1483

Last Page

1498

ISSN

1520-9210

Identifier

10.1109/TMM.2025.3639945

Publisher

Institute of Electrical and Electronics Engineers

Additional URL

https://doi.org/10.1109/TMM.2025.3639945

This document is currently not available here.

Share

COinS