Publication Type
Conference Proceeding Article
Version
acceptedVersion
Publication Date
11-2021
Abstract
Unifying acoustic and linguistic representation learning has become increasingly crucial to transfer the knowledge learned on the abundance of high-resource language data for low-resource speech recognition. Existing approaches simply cascade pre-trained acoustic and language models to learn the transfer from speech to text. However, how to solve the representation discrepancy of speech and text is unexplored, which hinders the utilization of acoustic and linguistic information. Moreover, previous works simply replace the embedding layer of the pre-trained language model with the acoustic features, which may cause the catastrophic forgetting problem. In this work, we introduce Wav-BERT, a cooperative acoustic and linguistic representation learning method to fuse and utilize the contextual information of speech and text. Specifically, we unify a pre-trained acoustic model (wav2vec 2.0) and a language model (BERT) into an end-to-end trainable framework. A Representation Aggregation Module is designed to aggregate acoustic and linguistic representation, and an Embedding Attention Module is introduced to incorporate acoustic information into BERT, which can effectively facilitate the cooperation of two pre-trained models and thus boost the representation learning. Extensive experiments show that our Wav-BERT significantly outperforms the existing approaches and achieves state-of-the-art performance on low-resource speech recognition.
Discipline
Graphics and Human Computer Interfaces | Programming Languages and Compilers
Research Areas
Intelligent Systems and Optimization
Areas of Excellence
Digital transformation
Publication
Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, Virtual Conference, November 7-11
First Page
2765
Last Page
2777
ISBN
9781955917094
Identifier
10.18653/V1/2021.FINDINGS-EMNLP.236
Publisher
ACL
City or Country
Punta Cana
Citation
ZHENG, Guolin; XIAO, Yubei; GONG, Ke; ZHOU, Pan; LIANG, Xiaodan; and LIN, Liang.
Wav-BERT: Cooperative acoustic and linguistic representation learning for low-resource speech recognition. (2021). Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, Virtual Conference, November 7-11. 2765-2777.
Available at: https://ink.library.smu.edu.sg/sis_research/9000
Creative Commons License
This work is licensed under a Creative Commons Attribution-NonCommercial-No Derivative Works 4.0 International License.
Additional URL
https://doi.org/10.18653/V1/2021.FINDINGS-EMNLP.236
Included in
Graphics and Human Computer Interfaces Commons, Programming Languages and Compilers Commons