Publication Type

Conference Proceeding Article

Version

publishedVersion

Publication Date

8-2024

Abstract

Conventional machine learning methods are predominantly designed to predict outcomes based on a single data type. However, practical applications may encompass data of diverse types, such as text, images, and audio. We introduce interpretable tensor fusion (InTense), a multimodal learning method training a neural network to simultaneously learn multiple data representations and their interpretable fusion. InTense can separately capture both linear combinations and multiplicative interactions of the data types, thereby disentangling higher-order interactions from the individual effects of each modality. InTense provides interpretability out of the box by assigning relevance scores to modalities and their associations, respectively. The approach is theoretically grounded and yields meaningful relevance scores on multiple synthetic and real-world datasets. Experiments on four real-world datasets show that InTense outperforms existing state-of-the-art multimodal interpretable approaches in terms of accuracy and interpretability.

Keywords

Tensor Fusion, Multiple Kernel Learning, Interpretability

Discipline

Databases and Information Systems

Research Areas

Data Science and Engineering

Publication

Proceedings of the Thirty-Third International Joint Conference on Artificial Intelligence, Jeju, South Korea, 2024 August 3-9

First Page

5037

Last Page

5045

Identifier

10.24963/ijcai.2024/557

Publisher

IJCAI

City or Country

Jeju

Additional URL

https://doi.org/10.24963/ijcai.2024/557

Download

Share

COinS