Title

Evaluating Defect Prediction Using A Massive Set of Metrics

Publication Type

Conference Proceeding Article

Publication Date

4-2015

Abstract

To evaluate the performance of a within-project defect prediction approach, people normally use precision, recall, and F-measure scores. However, in machine learning literature, there are a large number of evaluation metrics to evaluate the performance of an algorithm, (e.g., Matthews Correlation Coefficient, G-means, etc.), and these metrics evaluate an approach from different aspects. In this paper, we investigate the performance of within-project defect prediction approaches on a large number of evaluation metrics. We choose 6 state-of-the-art approaches including naive Bayes, decision tree, logistic regression, kNN, random forest and Bayesian network which are widely used in defect prediction literature. And we evaluate these 6 approaches on 14 evaluation metrics (e.g., G-mean, F-measure, balance, MCC, J-coefficient, and AUC). Our goal is to explore a practical and sophisticated way for evaluating the prediction approaches comprehensively. We evaluate the performance of defect prediction approaches on 10 defect datasets from PROMISE repository. The results show that Bayesian network achieves a noteworthy performance. It achieves the best recall, FN-R, G-mean1 and balance on 9 out of the 10 datasets, and F-measure and J-coefficient on 7 out of the 10 datasets.

Discipline

Software Engineering

Research Areas

Software and Cyber-Physical Systems

Publication

30th ACM Symposium on Applied Computing (SAC)

First Page

1644

Last Page

1647

ISBN

9781450331968

Identifier

10.1145/2695664.2695959

City or Country

Salamanca, Spain

Additional URL

http://dx.doi.org/10.1145/2695664.2695959

This document is currently not available here.

Share

COinS