LLMs-based augmentation for domain adaptation in long-tailed food datasets

Author

Qing WANG, Singapore Management UniversityFollow
Chong-wah NGO, Singapore Management UniversityFollow
Ee-peng LIM, Singapore Management UniversityFollow
Qianru SUN, Singapore Management UniversityFollow

Publication Type

Conference Proceeding Article

Version

acceptedVersion

Publication Date

1-2025

Abstract

Training a model for food recognition is challenging because the training samples, which are typically crawled from the Internet, are visually different from the pictures captured by users in the free-living environment. In addition to this domain-shift problem, the real-world food datasets tend to be long-tailed distributed and some dishes of different categories exhibit subtle variations that are difficult to distinguish visually. In this paper, we present a framework empowered with large language models (LLMs) to address these challenges in food recognition. We first leverage LLMs to parse food images to generate food titles and ingredients. Then, we project the generated texts and food images from different domains to a shared embedding space to maximize the pair similarities. Finally, we take the aligned features of both modalities for recognition. With this simple framework, we show that our proposed approach can outperform the existing approaches tailored for long-tailed data distribution, domain adaptation, and fine-grained classification, respectively, on two food datasets.

Keywords

Class imbalance, Domain adaptation, Food recognition

Discipline

Artificial Intelligence and Robotics | Databases and Information Systems

Research Areas

Data Science and Engineering

Publication

MultiMedia Modeling: 31st International Conference on Multimedia Modeling, MMM 2025, Nara, January 8-10, Proceedings

Volume

15521

First Page

282

Last Page

295

ISBN

9789819620609

Identifier

10.1007/978-981-96-2061-6_21

Publisher

Springer

City or Country

Cham

Citation

WANG, Qing; NGO, Chong-wah; LIM, Ee-peng; and SUN, Qianru. LLMs-based augmentation for domain adaptation in long-tailed food datasets. (2025). MultiMedia Modeling: 31st International Conference on Multimedia Modeling, MMM 2025, Nara, January 8-10, Proceedings. 15521, 282-295.
Available at: https://ink.library.smu.edu.sg/sis_research/10103

Creative Commons License

This work is licensed under a Creative Commons Attribution-NonCommercial-No Derivative Works 4.0 International License.

Additional URL

https://doi.org/10.1007/978-981-96-2061-6_21