Publication Type

Conference Proceeding Article

Version

publishedVersion

Publication Date

3-2015

Abstract

During software maintenance, testing is a crucial activity to ensure the quality of program code as it evolves over time. With the increasing size and complexity of software, adequate software testing has become increasingly important. Code coverage is often used as a yardstick to gauge the comprehensiveness of test cases and the adequacy of testing. A test suite quality is often measured by the number of bugs it can find (aka. kill). Previous studies have analysed the quality of a test suite by its ability to kill mutants, i.e., artificially seeded faults. However, mutants do not necessarily represent real bugs. Moreover, many studies use small programs which increases the threat of the applicability of the results on large real-world systems. In this paper, we analyse two large software systems to measure the relationship of code coverage and its effectiveness in killing real bugs from the software systems. We use Randoop, a random test generation tool to generate test suites with varying levels of coverage and run them to analyse if the test suites can kill each of the real bugs or not. In this preliminary study, we have performed an experiment on 67 and 92 real bugs from Apache HTTPClient and Mozilla Rhino, respectively. Our experiment finds that there is indeed statistically significant correlation between code coverage and bug kill effectiveness. The strengths of the correlation, however, differ for the two software systems. For HTTPClient, the correlation is moderate for both statement and branch coverage. For Rhino, the correlation is strong for both statement and branch coverage.

Discipline

Software Engineering

Research Areas

Software and Cyber-Physical Systems

Publication

22nd IEEE International Conference on Software Analysis, Evolution, and Reengineering (SANER)

First Page

560

Last Page

564

Identifier

10.1109/SANER.2015.7081877

Publisher

IEEE

City or Country

Montreal, Canada

Additional URL

http://dx.doi.org/10.1109/SANER.2015.7081877

Download

Share

COinS