Publication Type

Conference Proceeding Article

Version

acceptedVersion

Publication Date

10-2023

Abstract

Automated program repair (APR) has been gaining ground with substantial effort devoted to the area, opening up many challenges and opportunities. One such challenge is that the state-of-the-art repair techniques often resort to incomplete specifications, e.g., test cases that witness buggy behavior, to generate repairs. In practice, bug-exposing test cases are often available when: (1) developers, at the same time of (or after) submitting bug fixes, create the tests to assure the correctness of the fixes, or (2) regression errors occur. The former case – a scenario commonly used for creating popular bug datasets – however, may not be suitable to assess how APR performs in the wild. Since developers already know where and how to fix the bugs, tests created in this case may encapsulate knowledge gained only after bugs are fixed. Thus, more effort is needed to create datasets for more realistically evaluating APR.We address this challenge by creating a dataset focusing on bugs identified via continuous integration (CI) failures – a special case of regression errors – wherein bugs happen when the program after being changed is re-executed on the existing test suite. We argue that CI failures, wherein bug-exposing tests are created before bug fixes and thus assume no prior knowledge of developers on the bugs to be involved, are more realistic for evaluating APR. Toward this end, we curated 102 CI failures from 40 popular real-world software on GitHub. We demonstrate various features and the usefulness of the dataset via an evaluation of five well-known APR techniques, namely GenProg, Kali, Cardumen, RsRepair and Arja. We subsequently discuss several findings and implications for future APR studies. Overall, experiment results show that our dataset is complementary to existing datasets such as Defect4J in realistic evaluations of APR.

Keywords

Benchmark, Program Repair

Discipline

Software Engineering

Research Areas

Software and Cyber-Physical Systems

Publication

2023 IEEE International Conference on Software Maintenance and Evolution, ICSME: Bogota, October 1-6: Proceedings

First Page

50

Last Page

61

ISBN

9798350327830

Identifier

10.1109/ICSME58846.2023.00017

Publisher

IEE

City or Country

Piscataway, NJ

Additional URL

https://doi.org/10.1109/ICSME58846.2023.00017

Download

Share

COinS