Publication Type
Conference Proceeding Article
Version
submittedVersion
Publication Date
12-2023
Abstract
The success of large language models (LLMs), like GPT-4 and ChatGPT, has led to the development of numerous cost-effective and accessible alternatives that are created by finetuning open-access LLMs with task-specific data (e.g., ChatDoctor) or instruction data (e.g., Alpaca). Among the various fine-tuning methods, adapter-based parameter-efficient fine-tuning (PEFT) is undoubtedly one of the most attractive topics, as it only requires fine-tuning a few external parameters instead of the entire LLMs while achieving comparable or even better performance. To enable further research on PEFT methods of LLMs, this paper presents LLMAdapters, an easy-to-use framework that integrates various adapters into LLMs and can execute these adapter-based PEFT methods of LLMs for different tasks. The framework includes state-of-the-art open-access LLMs such as LLaMA, BLOOM, and GPT-J, as well as widely used adapters such as Series adapters, Parallel adapter, Prompt-based learning and Reparametrization-based methods. Moreover, we conduct extensive empirical studies on the impact of adapter types, placement locations, and hyper-parameters to the best design for each adapter-based methods. We evaluate the effectiveness of the adapters on fourteen datasets from two different reasoning tasks, Arithmetic Reasoning and Commonsense Reasoning. The results demonstrate that using adapter-based PEFT in smaller-scale LLMs (7B) with few extra trainable parameters yields comparable, and in some cases superior, performance to powerful LLMs (175B) in zero-shot inference on both reasoning tasks.
Discipline
Databases and Information Systems | Programming Languages and Compilers
Research Areas
Data Science and Engineering
Publication
Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing (EMNLP), Singapore, December 6-10
First Page
1
Last Page
21
Publisher
EMNLP
City or Country
Singapore
Citation
HU, Zhiqiang; WANG, Lei; LAN, Yihuai; XU, Wanyu; LIM, Ee-peng; BING, Lidong; XU, Xing; PORIA, Soujanya; and LEE, Roy Ka-Wei.
LLM-adapters: An adapter family for parameter-efficient fine-tuning of large language models. (2023). Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing (EMNLP), Singapore, December 6-10. 1-21.
Available at: https://ink.library.smu.edu.sg/sis_research/8324
Creative Commons License
This work is licensed under a Creative Commons Attribution-NonCommercial-No Derivative Works 4.0 International License.