Publication Type
Journal Article
Version
publishedVersion
Publication Date
12-2015
Abstract
Severity levels, e.g., critical and minor, of bugs are often used to prioritize development efforts. Prior research efforts have proposed approaches to automatically assign the severity label to a bug report. All prior efforts verify the accuracy of their approaches using human-assigned bug reports data that is stored in software repositories. However, all prior efforts assume that such human-assigned data is reliable. Hence a perfect automated approach should be able to assign the same severity label as in the repository – achieving a 100% accuracy. Looking at duplicate bug reports (i.e., reports referring to the same problem) from three open-source software systems (OpenOffice, Mozilla, and Eclipse), we find that around 51 % of the duplicate bug reports have inconsistent human-assigned severity labels even though they refer to the same software problem. While our results do indicate that duplicate bug reports have unreliable severity labels, we believe that they send warning signals about the reliability of the full bug severity data (i.e., including non-duplicate reports). Future research efforts should explore if our findings generalize to the full dataset. Moreover, they should factor in the unreliable nature of the bug severity data. Given the unreliable nature of the severity data, classical metrics to assess the accuracy of models/learners should not be used for assessing the accuracy of approaches for automated assigning severity label. Hence, we propose a new approach to assess the performance of such models. Our new assessment approach shows that current automated approaches perform well – 77-86 % agreement with human-assigned severity labels.
Keywords
Bug report management, Data quality, Noise prediction, Performance evaluation, Severity prediction
Discipline
Computer Sciences | Information Security | Software Engineering
Research Areas
Software and Cyber-Physical Systems
Publication
Empirical Software Engineering
Volume
21
Issue
6
First Page
2298
Last Page
2323
ISSN
1382-3256
Identifier
10.1007/s10664-015-9409-1
Publisher
Springer Verlag (Germany)
Embargo Period
10-27-2017
Citation
TIAN, Yuan; ALI, Nasir; David LO; and HASSAN, Ahmed E..
On the unreliability of bug severity data. (2015). Empirical Software Engineering. 21, (6), 2298-2323.
Available at: https://ink.library.smu.edu.sg/sis_research/2859
Copyright Owner and License
Authors
Creative Commons License
This work is licensed under a Creative Commons Attribution-NonCommercial-No Derivative Works 4.0 International License.
Additional URL
http://doi.org/10.1007/s10664-015-9409-1