Publication Type

Conference Proceeding Article

Version

acceptedVersion

Publication Date

12-2023

Abstract

Memes are widely used to convey cultural and societal issues and have a significant impact on public opinion. However, little work has been done on understanding and explaining the semantics expressed in multimodal memes. To fill this research gap, we introduce MERMAID, a dataset consisting of 3,633 memes annotated with their entities and relations, and propose a novel MERF pipeline that extracts entities and their relationships in memes. Our framework combines state-of-the-art techniques from natural language processing and computer vision to extract text and image features and infer relationships between entities in memes. We evaluate the proposed framework on a real-world meme dataset and establish the benchmark for the new multimodal meme semantic understanding task. Our evaluation also includes a low-resource setting, where we assess the applicability of our framework to low-resource settings, which is a common problem due to the high cost and lack of labeled data for relations in memes. Overall, our work contributes to the understanding of the semantics of memes, a crucial form of communication in today's society.

Keywords

Memes, Multimodal, Semantic Extraction

Discipline

Databases and Information Systems | Graphics and Human Computer Interfaces | Mental and Social Health

Publication

2023 IEEE International Conference on Big Data: Sorrento, Italy, December 15-18: Proceedings

First Page

433

Last Page

442

ISBN

9798350324457

Identifier

10.1109/BigData59044.2023.10386279

Publisher

IEEE

City or Country

Piscataway, NJ

Additional URL

https://doi.org/10.1109/BigData59044.2023.1038627

Download

Share

COinS