Publication Type
Conference Proceeding Article
Version
acceptedVersion
Publication Date
5-2021
Abstract
Recently, there has been a significant growth of interest in applying software engineering techniques for the quality assurance of deep learning (DL) systems. One popular direction is deep learning testing, where adversarial examples (a.k.a. bugs) of DL systems are found either by fuzzing or guided search with the help of certain testing metrics. However, recent studies have revealed that the commonly used neuron coverage metrics by existing DL testing approaches are not correlated to model robustness. It is also not an effective measurement on the confidence of the model robustness after testing. In this work, we address this gap by proposing a novel testing framework called Robustness-Oriented Testing (RobOT). A key part of RobOT is a quantitative measurement on 1) the value of each test case in improving model robustness (often via retraining), and 2) the convergence quality of the model robustness improvement. RobOT utilizes the proposed metric to automatically generate test cases valuable for improving model robustness. The proposed metric is also a strong indicator on how well robustness improvement has converged through testing. Experiments on multiple benchmark datasets confirm the effectiveness and efficiency of RobOT in improving DL model robustness, with 67.02% increase on the adversarial robustness that is 50.65% higher than the state-of-the-art work DeepGini.
Keywords
Measurement, Deep learning, Benchmark testing, Robustness, Robots, Software engineering, Convergence
Discipline
Numerical Analysis and Scientific Computing | Software Engineering
Research Areas
Software and Cyber-Physical Systems
Publication
2021 IEEE/ACM 43rd International Conference on Software Engineering (ICSE): Proceedings: Virtual, May 22-30
First Page
300
Last Page
311
ISBN
9781665402965
Identifier
10.1109/ICSE43902.2021.00038
Publisher
IEEE
City or Country
Piscataway, NJ
Embargo Period
8-25-2021
Citation
WANG, Jingyi; CHEN, Jialuo; SUN, Youcheng; MA, Xingjun; WANG, Dongxia; SUN, Jun; and CHENG, Peng.
RobOT: Robustness-oriented testing for deep learning systems. (2021). 2021 IEEE/ACM 43rd International Conference on Software Engineering (ICSE): Proceedings: Virtual, May 22-30. 300-311.
Available at: https://ink.library.smu.edu.sg/sis_research/6056
Copyright Owner and License
Authors
Creative Commons License
This work is licensed under a Creative Commons Attribution-NonCommercial-No Derivative Works 4.0 International License.
Additional URL
https://doi.org/10.1109/ICSE43902.2021.00038