Publication Type
Conference Proceeding Article
Version
acceptedVersion
Publication Date
10-2020
Abstract
Automatic speech recognition (ASR) systems are ubiquitous parts of modern life. It can be found in our smartphones, desktops, and smart home systems. To ensure its correctness in recognizing speeches, ASR needs to be tested. Testing ASR requires test cases in the form of audio files and their transcribed texts. Building these test cases manually, however, is tedious and time-consuming.To deal with the aforementioned challenge, in this work, we propose CrossASR, an approach that capitalizes the existing Text-To-Speech (TTS) systems to automatically generate test cases for ASR systems. CrossASR is a differential testing solution that compares outputs of multiple ASR systems to uncover erroneous behaviors among ASRs. CrossASR efficiently generates test cases to uncover failures with as few generated tests as possible; it does so by employing a failure probability predictor to pick the texts with the highest likelihood of leading to failed test cases. As a black-box approach, CrossASR can generate test cases for any ASR, including when the ASR model is not available (e.g., when evaluating the reliability of various third-party ASR services).We evaluated CrossASR using 4 TTSes and 4 ASRs on the Europarl corpus. The experimented ASRs are Deepspeech, Deepspeech2, wav2letter, and wit. Our experiments on a randomly sampled 20,000 English texts showed that within an hour, CrossASR can produce, on average from 3 experiments, 130.34, 123.33, 47.33, and 8.66 failed test cases using Google, Respon-siveVoice, Festival, and Espeak TTSes, respectively. Moreover, when we run CrossASR on the entire 20,000 texts, it can generate 13,572, 13,071, 5,911, and 1,064 failed test cases using Google, ResponsiveVoice, Festival, and Espeak TTSes, respectively. Based on a manual verification carried out on statistically representative sample size, we found that most samples are actual failed test cases (audio understandable to humans but cannot be transcribed properly by an ASR), demonstrating that CrossASR is highly reliable in determining failed test cases. We also make the source code for CrossASR and evaluation data available at https://github.com/soarsμCrossASR.
Keywords
Automatic Speech Recognition, Differential Testing, Failure Probability Predictor, Test Case Generation, Text-to-Speech
Discipline
Software Engineering
Research Areas
Software and Cyber-Physical Systems
Publication
2020 36th IEEE International Conference on Software Maintenance and Evolution (ICSME): 27 Sep - 3 Oct, Adelaide, Australia: Proceedings
First Page
640
Last Page
650
ISBN
9781728156194
Identifier
10.1109/ICSME46990.2020.00066
Publisher
IEEE
City or Country
Piscataway, NJ
Citation
ASYROFI, Muhammad Hilmi; Ferdian, Thung; LO, David; and JIANG, Lingxiao.
CrossASR: Efficient differential testing of automatic speech recognition via text-to-speech. (2020). 2020 36th IEEE International Conference on Software Maintenance and Evolution (ICSME): 27 Sep - 3 Oct, Adelaide, Australia: Proceedings. 640-650.
Available at: https://ink.library.smu.edu.sg/sis_research/5536
Copyright Owner and License
Authors
Creative Commons License
This work is licensed under a Creative Commons Attribution-NonCommercial-No Derivative Works 4.0 International License.
Additional URL
https://doi.org/10.1109/ICSME46990.2020.00066