Publication Type
Journal Article
Version
acceptedVersion
Publication Date
12-2022
Abstract
Recently, there has been a significant growth of interest in applying software engineering techniques for the quality assurance of deep learning (DL) systems. One popular direction is deep learning testing, i.e., given a property of test, defects of DL systems are found either by fuzzing or guided search with the help of certain testing metrics. However, recent studies have revealed that the neuron coverage metrics, commonly used by most existing DL testing approaches, are not necessarily correlated with model quality (e.g., robustness, the most studied model property), and are also not an effective measurement on the confidence of the model quality after testing. In this work, we address this gap by proposing a novel testing framework called QuoTe (i.e., Quality-oriented Testing). A key part of QuoTe is a quantitative measurement on 1) the value of each test case in enhancing the model property of interest (often via retraining), and 2) the convergence quality of the model property improvement. QuoTe utilizes the proposed metric to automatically select or generate valuable test cases for improving model quality. The proposed metric is also a lightweight yet strong indicator of how well the improvement converged. Extensive experiments on both image and tabular datasets with a variety of model architectures confirm the effectiveness and efficiency of QuoTe in improving DL model quality, i.e., robustness and fairness. As a generic quality-oriented testing framework, future adaptions can be made to other domains (e.g., text) as well as other model properties.
Keywords
Deep learning, testing, robustness, fairness, software debugging
Discipline
Software Engineering
Research Areas
Software and Cyber-Physical Systems
Publication
ACM Transactions on Software Engineering and Methodology
Volume
32
Issue
5
First Page
1
Last Page
33
ISSN
1049-331X
Identifier
10.1145/3582573
Publisher
Association for Computing Machinery (ACM)
Citation
CHEN, Jialuo; WANG, Jingyi; MA, Xingjun; SUN, Youcheng; SUN, Jun; ZHANG, Peixin; and CHENG, Peng.
QuoTe: Quality-oriented Testing for deep learning systems. (2022). ACM Transactions on Software Engineering and Methodology. 32, (5), 1-33.
Available at: https://ink.library.smu.edu.sg/sis_research/7785
Copyright Owner and License
Authors
Creative Commons License
This work is licensed under a Creative Commons Attribution-NonCommercial-No Derivative Works 4.0 International License.
Additional URL
https://doi.org/10.1145/3582573