Publication Type

Journal Article

Version

publishedVersion

Publication Date

4-2025

Abstract

As Automated Speech Recognition (ASR) systems gain widespread acceptance, there is a pressing need to rigorously test and enhance their performance. Nonetheless, the process of collecting and executing speech test cases is typically both costly and time-consuming. This presents a compelling case for the strategic prioritization of speech test cases, which consist of a piece of audio and the corresponding reference text. The central question we address is: In what sequence should speech test cases be collected and executed to identify the maximum number of errors at the earliest stage? In this study, we introduce PRiOritizing sPeecH tEsT (Prophet) cases, a tool designed to predict the likelihood that speech test cases will identify errors. Consequently, Prophet can assess and prioritize these test cases without having to run the ASR system, facilitating large-scale analysis. Our evaluation encompasses  distinct prioritization techniques across  ASR systems and  datasets. When constrained by the same test budget, our approach identified  more misrecognized words than the leading state-of-the-art method. We select top-ranked speech test cases from the prioritized list to fine-tune ASR systems and analyze how our approach can improve the ASR system performance. Statistical evaluations show that our method delivers a considerably higher performance boost for ASR systems compared to established baseline techniques. Moreover, our correlation analysis confirms that fine-tuning an ASR system with a dataset where the model initially underperforms tends to yield greater performance improvements.

Keywords

Automated Speech Recognition, DNN Model Quality, Test Case prioritization

Discipline

Numerical Analysis and Scientific Computing | Software Engineering

Research Areas

Software and Cyber-Physical Systems

Publication

ACM Transactions on Software Engineering and Methodology

Volume

34

Issue

4

First Page

1

Last Page

27

ISSN

1049-331X

Identifier

10.1145/3707450

Publisher

ACM

Embargo Period

4-13-2026

Copyright Owner and License

Authors

Creative Commons License

Creative Commons Attribution-No Derivative Works 4.0 International License
This work is licensed under a Creative Commons Attribution-No Derivative Works 4.0 International License.

Additional URL

https://doi.org/10.1145/3707450

Download
Download Research Data

Share

COinS