Publication Type

Conference Proceeding Article

Version

acceptedVersion

Publication Date

4-2021

Abstract

In this era of multimedia Web, text-to-image retrieval is a critical function of search engines and visually-oriented online platforms. Traditionally, the task primarily deals with matching a text query with the most relevant images available in the corpus. To an increasing extent, the Web also features visual expressions of preferences, imbuing images with sentiments that express those preferences. Cases in point include photos in online reviews as well as social media. In this work, we study the effects of sentiment information on text-to-image retrieval. Particularly, we present two approaches for incorporating sentiment orientation into metric learning for cross-modal retrieval. Each model emphasizes a hypothesis on how positive and negative sentiment vectors may be aligned in the metric space that also includes text and visual vectors. Comprehensive experiments and analyses on Visual Sentiment Ontology (VSO) and Yelp.com online reviews datasets show that our models significantly boost the retrieval performance as compared to various sentiment-insensitive baselines.

Keywords

Text-to-image retrieval, Cross-modal retrieval, Metric learning, Sentiment orientation

Discipline

Databases and Information Systems | Data Science | Numerical Analysis and Scientific Computing

Research Areas

Data Science and Engineering

Publication

Advances in Information Retrieval: 43rd European Conference on IR Research ECIR 2021, Virtual, March 28 - April 1: Proceedings

Volume

12656

First Page

634

Last Page

649

ISBN

9783030721121

Identifier

10.1007/978-3-030-72113-8_42

Publisher

Springer

City or Country

Cham

Embargo Period

5-20-2021

Copyright Owner and License

Authors

Additional URL

https://doi.org/10.1007/978-3-030-72113-8_42

Download

Share

COinS