Publication Type
Journal Article
Version
publishedVersion
Publication Date
3-2025
Abstract
Software development involves collaborative interactions where stakeholders express opinions across various platforms. Recognizing the sentiments conveyed in these interactions is crucial for the effective development and ongoing maintenance of software systems. For software products, analyzing the sentiment of user feedback, e.g., reviews, comments, and forum posts can provide valuable insights into user satisfaction and areas for improvement. This can guide the development of future updates and features. However, accurately identifying sentiments in software engineering datasets remains challenging.This study investigates bigger large language models (bLLMs) in addressing the labeled data shortage that hampers fine-tuned smaller large language models (sLLMs) in software engineering tasks. We conduct a comprehensive empirical study using five established datasets to assess three open source bLLMs in zero-shot and few-shot scenarios. Additionally, we compare them with fine-tuned sLLMs, using sLLMs to learn contextual embeddings of text from software platforms.Our experimental findings demonstrate that bLLMs exhibit state-of-the-art performance on datasets marked by limited training data and imbalanced distributions. bLLMs can also achieve excellent performance under a zero-shot setting. However, when ample training data are available or the dataset exhibits a more balanced distribution, fine-tuned sLLMs can still achieve superior results.
Keywords
Large Language Models, Sentiment Analysis, Software Engineering
Discipline
Artificial Intelligence and Robotics | Software Engineering
Research Areas
Software and Cyber-Physical Systems
Publication
ACM Transactions on Software Engineering and Methodology
Volume
34
Issue
3
First Page
1
Last Page
30
ISSN
1049-331X
Identifier
10.1145/3697009
Publisher
Association for Computing Machinery (ACM)
Citation
ZHANG, Ting; IRSAN, Ivana Clairine; Ferdian, Thung; and LO, David.
Revisiting sentiment analysis for software engineering in the era of large language models. (2025). ACM Transactions on Software Engineering and Methodology. 34, (3), 1-30.
Available at: https://ink.library.smu.edu.sg/sis_research/10224
Copyright Owner and License
Authors-CC-BY
Creative Commons License
This work is licensed under a Creative Commons Attribution 3.0 License.
Additional URL
https://doi.org/10.1145/36970