Publication Type
Journal Article
Version
acceptedVersion
Publication Date
3-2023
Abstract
Over the past few years, SMT string solvers have found their applications in an increasing number of domains, such as program analyses in mobile and Web applications, which require the ability to reason about string values. A series of research has been carried out to find quality issues of string solvers in terms of its correctness and performance. Yet, none of them has considered the performance regressions happening across multiple versions of a string solver. To fill this gap, in this paper, we focus on solver performance regressions (SPRs), i.e., unintended slowdowns introduced during the evolution of string solvers. To this end, we develop SPRFinder to not only generate test cases demonstrating SPRs, but also localize the probable causes of them, in terms of commits. We evaluated the effectiveness of SPRFinder on three state-of-the-art string solvers, i.e., Z3Seq, Z3Str3, and CVC4. The results demonstrate that SPRFinder is effective in generating SPR-inducing test cases and also able to accurately locate the responsible commits. Specifically, the average running time on the target versions is 13.2 slower than that of the reference versions. Besides, we also conducted the first empirical study to peek into the characteristics of SPRs, including the impact of random seed configuration for SPR detection, understanding the root causes of SPRs, and characterizing the regression test cases through case studies.
Keywords
Computer Bugs, Testing, Fuzzing, Codes, Runtime, Location Awareness, Cognition, SMT String Solver, Performance Regression, SPR Finder
Discipline
Software Engineering
Research Areas
Software and Cyber-Physical Systems
Publication
IEEE Transactions on Software Engineering
Volume
49
Issue
3
First Page
947
Last Page
961
ISSN
0098-5589
Identifier
10.1109/TSE.2022.3168373
Publisher
Institute of Electrical and Electronics Engineers
Citation
ZHANG, Yao; XIE, Xiaofei; LI, Yi; LIN, Yi; CHEN, Sen; LIU, Yang; and LI, Xiaohong.
Demystifying performance regressions in string solvers. (2023). IEEE Transactions on Software Engineering. 49, (3), 947-961.
Available at: https://ink.library.smu.edu.sg/sis_research/7239
Copyright Owner and License
Authors
Creative Commons License
This work is licensed under a Creative Commons Attribution-NonCommercial-No Derivative Works 4.0 International License.
Additional URL
https://doi.org/10.1109/TSE.2022.3168373