Publication Type

Journal Article

Version

acceptedVersion

Publication Date

5-2024

Abstract

Vision-Language Pre-Training (VLP) has demonstrated remarkable potential in aligning image and text pairs, paving the way for a wide range of cross-modal learning tasks. Nevertheless, we have observed that VLP models often fall short in terms of visual grounding and localization capabilities, which are crucial for many downstream tasks, such as visual reasoning. In response, we introduce a novel Position-guided Text Prompt ( PTP ) paradigm to bolster the visual grounding abilities of cross-modal models trained with VLP. In the VLP phase, PTP divides an image into N x N blocks and employs a widely-used object detector to identify objects within each block. PTP then reframes the visual grounding task as a fill-in-the-blank problem, encouraging the model to predict objects in given blocks or regress the blocks of a given object, exemplified by filling “ [P] ” or “ [O] ” in a PTP sentence such as “ The block [P] has a [O]. ” This strategy enhances the visual grounding capabilities of VLP models, enabling them to better tackle various downstream tasks. Additionally, we integrate the seconda-order relationships between objects to further enhance the visual grounding capabilities of our proposed PTP paradigm. Incorporating PTP into several state-of-the-art VLP frameworks leads to consistently significant improvements across representative cross-modal learning model architectures and multiple benchmarks, such as zero-shot Flickr30 k Retrieval (+5.6 in average recall@1) for ViLT baseline, and COCO Captioning (+5.5 in CIDEr) for the state-of-the-art BLIP baseline. Furthermore, PTP attains comparable results with object-detector-based methods and a faster inference speed, as it discards its object detector during inference, unlike other approaches.

Keywords

Fill-in-the-blank, position-guided text prompt, vision-language pre-training, visual grounding

Discipline

Artificial Intelligence and Robotics | Numerical Analysis and Scientific Computing | Programming Languages and Compilers

Research Areas

Intelligent Systems and Optimization

Publication

IEEE Transactions on Pattern Analysis and Machine Intelligence

Volume

46

Issue

5

First Page

3406

Last Page

3421

ISSN

0162-8828

Identifier

10.1109/TPAMI.2023.3343736

Publisher

Institute of Electrical and Electronics Engineers

Copyright Owner and License

Authors

Additional URL

https://doi.org/10.1109/TPAMI.2023.3343736

Download

Share

COinS