Publication Type

Conference Proceeding Article

Version

publishedVersion

Publication Date

5-2022

Abstract

Manually finding the program unit (e.g., class, method, or statement) responsible for a fault is tedious and time-consuming. To mitigate this problem, many fault localization techniques have been proposed. A popular family of such techniques is spectrum-based fault localization (SBFL), which takes program execution traces (spectra) of failed and passed test cases as input and applies a ranking formula to compute a suspiciousness score for each program unit. However, most existing SBFL techniques fail to consider two facts: 1) not all failed test cases contribute equally to a considered fault(s), and 2) program units collaboratively contribute to the failure/pass of each test case in different ways. In this study, we propose a novel idea that first models the SBFL task as a classification problem of predicting whether a test case will fail or pass based on spectra information on program units. We subsequently apply eXplainable Artificial Intelligence (XAI) techniques to infer the local importance of each program unit to the prediction of each executed test case. Applying XAI to the failed test case, we retrieve information about which program statements within the test case that are considered the most important (i.e., have the biggest effect in making the test case failed). Such a design can automatically learn the unique contributions of failed test cases to the suspiciousness of a program unit by learning the different and collaborative contributions of program units to each test case's executed result. As far as we know, this is the first XAI-supported SBFL approach. We evaluate the new approach on the Defects4J benchmark dataset. We compare the performance of our approach against five popular SBFL techniques: DStar, Tarantula, Barinel, Ochiai, and OP. We measure their performance using the Top-K and EXAM scores. In particular, we focus on the result of the Top-1, which importance has been highlighted in automated program repair domain, where the proposed methods often assume perfect fault localization (i.e., the fault must be found at the first rank of the suspiciousness list). Our results show that our approach, named XAI4FL, has a statistically significant and substantially better performance in terms of Top-1 than the SBFL approaches. We also compare our approach with a simpler approach to get feature importance in a tree-based model (i.e., using the Mean Decrease in Impurity method). Our results show that XAI4FL statistically significantly outperforms the MDI method in Top-K and EXAM score. Our results and findings highlight that the utilization of XAI for fault localization can improve the overall results of fault localization techniques.

Keywords

Fault localization, Explainable artificial intelligence (XAI), Model-agnostic explanation technique, Spectrum-based fault localization, Testing and debugging

Discipline

Software Engineering

Research Areas

Intelligent Systems and Optimization

Publication

Proceedings of the 30th IEEE/ACM International Conference on Program Comprehension, Pittsburgh, USA, 2022 May 16-17

First Page

499

Last Page

510

ISBN

9781450392983

Identifier

10.1145/3524610.3527902

Publisher

Institute of Electrical and Electronics Engineers Inc

City or Country

Pittsburgh, USA

Additional URL

https://doi.org/10.1145/3524610.3527902

Download

Share

COinS