Learning to match anchor-target video pairs with dual attentional holographic networks

Publication Type

Journal Article

Publication Date

1-2021

Abstract

Video hyperlinking is the task of linking two video fragments/clips based on their multi-modal contents. Specifically, given an anchor video as a query, machine techniques automatically generate links between the anchor and target videos by modeling and comparing their content aboutness. The term "aboutness" specifically refers to contextually relevant multimedia content, i.e., a fragment is on or of something. Since video contents are multi-modal (e.g., audio and vision), the content aboutness may be reflected across different modalities. Existing approaches regard hyperlinking as a retrieval task, by embedding multi-modal video contents into one or multiple common video representation space(s) for cross-modal comparison. As a result, the aboutness between videos is scored by computing the vector-distance based similarity in the learnt common feature space. However, these methods suffer from two main limitations: (1) the video modality descriptors/features are treated equally in representation learning, which hinders the effective modeling of their respective capabilities in linking; and (2) directly using the vector-distance based similarity to measure aboutness bears the risk of returning more duplicates. This paper focuses on addressing these two problems. Specifically, we firstly build attentional neural networks to learn a compact fragment-level representation, assigning different importance weights to different descriptor/feature contents by an attention mechanism. We believe that the potentially interesting content(s) should be highlighted in the representation. Furthermore, instead of directly computing the similarity of two representation embeddings, we secondly build a holographic composition network to model the aboutness for link establishment, with the core use of circular correlation. The two networks string together to form the final hyperlinking matching system. The entire model is trained in an end-to-end fashion. We examine its effectiveness by creating four train/validate/test partitioning schemes on the Blip10000 dataset and employing two video fragmentation methods.

Keywords

Task analysis, Neural networks, Correlation, Semantics, Generative adversarial networks, Feature extraction, Benchmark testing, Video hyperlinking, hierarchical content attention, holographic composition, triplet loss

Discipline

Graphics and Human Computer Interfaces | OS and Networks

Research Areas

Intelligent Systems and Optimization

Publication

IEEE Transactions on Image Processing

Volume

30

First Page

8130

Last Page

8143

ISSN

1057-7149

Identifier

10.1109/TIP.2021.3113165

Publisher

Institute of Electrical and Electronics Engineers

This document is currently not available here.

Share

COinS