Publication Type
Conference Proceeding Article
Version
publishedVersion
Publication Date
9-2023
Abstract
Learning-based techniques, especially advanced Large Language Models (LLMs) for code, have gained considerable popularity in various software engineering (SE) tasks. However, most existing works focus on designing better learning-based models and pay less attention to the properties of datasets. Learning-based models, including popular LLMs for code, heavily rely on data, and the data's properties (e.g., data distribution) could significantly affect their behavior. We conducted an exploratory study on the distribution of SE data and found that such data usually follows a skewed distribution (i.e., long-tailed distribution) where a small number of classes have an extensive collection of samples, while a large number of classes have very few samples. We investigate three distinct SE tasks and analyze the impacts of long-tailed distribution on the performance of LLMs for code. Our experimental results reveal that the long-tailed distribution has a substantial impact on the effectiveness of LLMs for code. Specifically, LLMs for code perform between 30.0% and 254.0% worse on data samples associated with infrequent labels compared to data samples of frequent labels. Our study provides a better understanding of the effects of long-tailed distributions on popular LLMs for code and insights for the future development of SE automation.
Keywords
Code distributions, Data distribution, Data properties, Data sample, Engineering tasks, Language model, Learning Based Models, Long-tailed distributions, Number of class, Property
Discipline
Databases and Information Systems | Software Engineering
Research Areas
Data Science and Engineering
Publication
Proceedings of the 38th IEEE/ACM International Conference on Automated Software Engineering, Echternach, Luxembourg, 2023 September 11-15
First Page
40
Last Page
52
ISBN
9798350329964
Identifier
10.1109/ASE56229.2023.00157
Publisher
IEEE
City or Country
New Jersey
Citation
ZHOU, Xin; KIM, Kisub; XU, Bowen; LIU, Jiakun; HAN, DongGyun; and LO, David.
The devil is in the tails: How long-tailed code distributions impact large language models. (2023). Proceedings of the 38th IEEE/ACM International Conference on Automated Software Engineering, Echternach, Luxembourg, 2023 September 11-15. 40-52.
Available at: https://ink.library.smu.edu.sg/sis_research/8568
Creative Commons License
This work is licensed under a Creative Commons Attribution-NonCommercial-No Derivative Works 4.0 International License.
Additional URL
https://doi.org/10.1109/ASE56229.2023.00157