Publication Type

Conference Proceeding Article

Version

publishedVersion

Publication Date

10-2021

Abstract

Video is complex due to large variations in motion and rich content in fine-grained visual details. Abstracting useful information from such information-intensive media requires exhaustive computing resources. This paper studies a two-step alternative that first condenses the video sequence to an informative" frame" and then exploits off-the-shelf image recognition system on the synthetic frame. A valid question is how to define" useful information" and then distill it from a video sequence down to one synthetic frame. This paper presents a novel Informative Frame Synthesis (IFS) architecture that incorporates three objective tasks, ie, appearance reconstruction, video categorization, motion estimation, and two regularizers, ie, adversarial learning, color consistency. Each task equips the synthetic frame with one ability, while each regularizer enhances its visual quality. With these, by jointly learning the frame synthesis in an end-to-end manner, the generated frame is expected to encapsulate the required spatio-temporal information useful for video analysis. Extensive experiments are conducted on the large-scale Kinetics dataset. When comparing to baseline methods that map video sequence to a single image, IFS shows superior performance. More remarkably, IFS consistently demonstrates evident improvements on image-based 2D networks and clip-based 3D networks, and achieves comparable performance with the state-of-the-art methods with less computational cost.

Discipline

Databases and Information Systems | Graphics and Human Computer Interfaces

Research Areas

Data Science and Engineering

Publication

Proceedings of the IEEE/CVF International Conference on Computer Vision, Virtual Online, October 11-17

First Page

16311

Last Page

16320

Publisher

IEEE

City or Country

New York

Download

Share

COinS