Publication Type
Journal Article
Version
acceptedVersion
Publication Date
12-2013
Abstract
In recent years, multimodal fusion has emerged as a promising technology for effective multimedia retrieval. Developing the optimal fusion strategy for different modality (e.g. content, metadata) has been the subject of intensive research. Given a query, existing methods derive a unified fusion strategy for all documents with the underlying assumption that the relative significance of a modality remains the same across all documents. However, this assumption is often invalid. We thus propose a general multimodal fusion framework, query-document-dependent fusion (QDDF), which derives the optimal fusion strategy for each query-document pair via intelligent content analysis of both queries and documents. By investigating multimodal fusion strategies adaptive to both queries and documents, we demonstrate that existing multimodal fusion approaches are special cases of QDDF and propose two QDDF approaches to derive fusion strategies. The dual-phase QDDF explicitly derives and fuses query- and document-dependent weights, and the regression-based QDDF determines the fusion weight for a query-document pair via a regression model derived from training data. To evaluate the proposed approaches, comprehensive experiments have been conducted using a multimedia data set with around 17K full songs and over 236K social queries. Results indicate that the regression-based QDDF is superior in handling single-dimension queries. In comparison, the dual-phase QDDF outperforms existing approaches for most query types. We found that document-dependent weights are instrumental in enhancing multimedia fusion performance. In addition, efficiency analysis demonstrates the scalability of QDDF over large data sets.
Keywords
Information retrieval, multimodal, query-document-dependent fusion
Discipline
Databases and Information Systems | Music
Research Areas
Data Science and Engineering
Publication
IEEE Transactions on Multimedia
Volume
15
Issue
8
First Page
1830
Last Page
1842
ISSN
1520-9210
Identifier
10.1109/TMM.2013.2280437
Publisher
Institute of Electrical and Electronics Engineers (IEEE)
Citation
LI, Zhonghua; ZHANG, Bingjun; YU, Yi; SHEN, Jialie; and WANG, Ye.
Query-document-dependent fusion: A case study of multimodal music retrieval. (2013). IEEE Transactions on Multimedia. 15, (8), 1830-1842.
Available at: https://ink.library.smu.edu.sg/sis_research/1822
Copyright Owner and License
Authors
Creative Commons License
This work is licensed under a Creative Commons Attribution-NonCommercial-No Derivative Works 4.0 International License.
Additional URL
https://doi.org/10.1109/TMM.2013.2280437