Publication Type
Master Thesis
Version
publishedVersion
Publication Date
9-2024
Abstract
Strong capabilities of generalization to unseen domains are vital for deep neural networks. While existing methods have shown promising results without source domain access, they mostly rely on models that are extensively pre-trained on source domains or overlook the intricate hierarchical structures inherent in visual and textual features. These limitations may have bad impacts on performances, especially on datasets with many classes. To overcome this, in this paper we propose a novel approach that projects the model onto hyperbolic geometry and employs geometric optimal transport to align cross-modal features in an unsupervised manner. Unlike Euclidean geometry, hyperbolic geometry is characterized by hierarchical data structures, which can facilitate understanding diverse classes. To fully capture hierarchical information from text, we enrich the model with finegrained concepts extracted from WordNet, enhancing its understanding of diverse classes. Extensive experiments on standard benchmarks demonstrate the superior performance of our method compared to strong baselines.
Keywords
Multi-modal Learning, Geometric deep learning, Vision-language model, Domain adaptation
Degree Awarded
MSc in Applied Finance (SUFE)
Discipline
Artificial Intelligence and Robotics
Supervisor(s)
CAO, Zhiguang; LIAO, Lizi
First Page
1
Last Page
48
Publisher
Singapore Management University
City or Country
Singapore
Citation
LIU, Suyu.
Multi-modal alignment via hyperbolic geometry. (2024). 1-48.
Available at: https://ink.library.smu.edu.sg/etd_coll/651
Copyright Owner and License
Author
Creative Commons License
This work is licensed under a Creative Commons Attribution-NonCommercial-No Derivative Works 4.0 International License.
