Aries: Efficient testing of deep neural networks via labeling-free accuracy estimation
Publication Type
Conference Proceeding Article
Publication Date
5-2023
Abstract
Deep learning (DL) plays a more and more important role in our daily life due to its competitive performance in industrial application domains. As the core of DL-enabled systems, deep neural networks (DNNs) need to be carefully evaluated to ensure the produced models match the expected requirements. In practice, the de facto standard to assess the quality of DNNs in the industry is to check their performance (accuracy) on a collected set of labeled test data. However, preparing such labeled data is often not easy partly because of the huge labeling effort, i.e., data labeling is labor-intensive, especially with the massive new incoming unlabeled data every day. Recent studies show that test selection for DNN is a promising direction that tackles this issue by selecting minimal representative data to label and using these data to assess the model. However, it still requires human effort and cannot be automatic. In this paper, we propose a novel technique, named Aries, that can estimate the performance of DNNs on new unlabeled data using only the information obtained from the original test data. The key insight behind our technique is that the model should have similar prediction accuracy on the data which have similar distances to the decision boundary. We performed a large-scale evaluation of our technique on two famous datasets, CIFAR-10 and Tiny-ImageNet, four widely studied DNN models including ResNetl0l and DenseNetl21, and 13 types of data transformation methods. Results show that the estimated accuracy by Aries is only 0.03% - 2.60% off the true accuracy. Besides, Aries also outperforms the state-of-the-art labeling-free methods in 50 out of 52 cases and selection-labeling-based methods in 96 out of 128 cases.
Keywords
Deep learning testing, Distribution shift, Performance estimation
Discipline
Artificial Intelligence and Robotics
Research Areas
Intelligent Systems and Optimization
Publication
Proceedings of the 45th International Conference on Software Engineering, Melbourne, Australia, May 14-20
First Page
1776
Last Page
1787
Identifier
10.1109/ICSE48619.2023.00152
City or Country
Manhattan, New York City, US
Citation
HU, Qiang; GUO, Yuejun; XIE, Xiaofei; CORDY, Maxime; MA, Lei; PAPADAKIS, Mike; and LE TRAON, Yves.
Aries: Efficient testing of deep neural networks via labeling-free accuracy estimation. (2023). Proceedings of the 45th International Conference on Software Engineering, Melbourne, Australia, May 14-20. 1776-1787.
Available at: https://ink.library.smu.edu.sg/sis_research/8243
Additional URL
https://doi.org/10.1109/ICSE48619.2023.00152