Cross-modal graph with meta concepts for video captioning

Publication Type

Journal Article

Publication Date

1-2022

Abstract

Video captioning targets interpreting the complex visual contents as text descriptions, which requires the model to fully understand video scenes including objects and their interactions. Prevailing methods adopt off-the-shelf object detection networks to give object proposals and use the attention mechanism to model the relations between objects. They often miss some undefined semantic concepts of the pretrained model and fail to identify exact predicate relationships between objects. In this paper, we investigate an open research task of generating text descriptions for the given videos, and propose Cross-Modal Graph (CMG) with meta concepts for video captioning. Specifically, to cover the useful semantic concepts in video captions, we weakly learn the corresponding visual regions for text descriptions, where the associated visual regions and textual words are named cross-modal meta concepts. We further build meta concept graphs dynamically with the learned cross-modal meta concepts. We also construct holistic video-level and local frame-level video graphs with the predicted predicates to model video sequence structures. We validate the efficacy of our proposed techniques with extensive experiments and achieve state-of-the-art results on two public datasets.

Keywords

Semantics, Visualization, Feature extraction, Predictive models, Task analysis, Computational modeling, Location awareness, Video captioning, vision-and-language

Discipline

Databases and Information Systems

Research Areas

Intelligent Systems and Optimization; Data Science and Engineering

Publication

IEEE Transactions on Image Processing

Volume

31

First Page

5150

Last Page

5162

ISSN

1057-7149

Identifier

10.1109/TIP.2022.3192709

Publisher

Institute of Electrical and Electronics Engineers

Additional URL

https://doi.org/10.1109/TIP.2022.3192709

This document is currently not available here.

Share

COinS