Publication Type

Conference Proceeding Article

Version

publishedVersion

Publication Date

9-2019

Abstract

We investigate the problem of object relationship classification of visual scenes. For a relationship object1-predicate-object2 that captures the object interaction, its representation is composed by the combination of object1 and object2 features. As a result, relationship classification models usually bias to the frequent objects, leading to poor generalization to rare or unseen objects. Inspired by the data augmentation methods, we propose a novel Semantic Transform Generative Adversarial Network (ST-GAN) that synthesizes relationship features for rare objects, conditioned on the features from random instances of the objects. Specifically, ST-GAN essentially offers a semantic transform function from cheap object features to expensive relationship features. Here, “cheap” means any easy-to-collect object which possesses an original but undesired relationship attribute, e.g., a sitting person; “expensive” means a target relationship on this object, e.g., person-riding-horse. By generating massive triplet combinations from any object pair with larger variance, ST-GAN can reduce the data bias. Extensive experiments on two benchmarks – Visual Relationship Detection (VRD) and Visual Genome (VG), show that using our synthesized features for data augmentation, the relationship classification model can be consistently improved in various settings such as zero-shot and low-shot.

Keywords

Visual relationship, object detection, feature generation

Discipline

Artificial Intelligence and Robotics | Numerical Analysis and Scientific Computing

Research Areas

Data Science and Engineering

Publication

BMVC 2019: 30th British Machine Vision Conference: Cardiff, September 9-12: Proceedings

First Page

1

Last Page

16

Publisher

BMVA Press

City or Country

Guildford

Copyright Owner and License

Authors

Additional URL

https://bmvc2019.org/wp-content/uploads/2019/09/bmvc_book_final.pdf

Download

Share

COinS