Publication Type

Conference Proceeding Article

Version

publishedVersion

Publication Date

4-2015

Abstract

To evaluate the performance of a within-project defect prediction approach, people normally use precision, recall, and F-measure scores. However, in machine learning literature, there are a large number of evaluation metrics to evaluate the performance of an algorithm, (e.g., Matthews Correlation Coefficient, G-means, etc.), and these metrics evaluate an approach from different aspects. In this paper, we investigate the performance of within-project defect prediction approaches on a large number of evaluation metrics. We choose 6 state-of-the-art approaches including naive Bayes, decision tree, logistic regression, kNN, random forest and Bayesian network which are widely used in defect prediction literature. And we evaluate these 6 approaches on 14 evaluation metrics (e.g., G-mean, F-measure, balance, MCC, J-coefficient, and AUC). Our goal is to explore a practical and sophisticated way for evaluating the prediction approaches comprehensively. We evaluate the performance of defect prediction approaches on 10 defect datasets from PROMISE repository. The results show that Bayesian network achieves a noteworthy performance. It achieves the best recall, FN-R, G-mean1 and balance on 9 out of the 10 datasets, and F-measure and J-coefficient on 7 out of the 10 datasets.

Keywords

Defect Prediction, Evaluation Metric, Machine Learning

Discipline

Software Engineering

Research Areas

Software and Cyber-Physical Systems

Publication

SAC '15: Proceedings of the 30th Annual ACM Symposium on Applied Computing, Salamanca, Spain, April 13-17

First Page

1644

Last Page

1647

ISBN

9781450331968

Identifier

10.1145/2695664.2695959

Publisher

ACM

City or Country

New York

Copyright Owner and License

Authors

Additional URL

https://doi.org/10.1145/2695664.2695959

Download

Share

COinS