Publication Type

Conference Proceeding Article

Version

acceptedVersion

Publication Date

11-2021

Abstract

Unifying acoustic and linguistic representation learning has become increasingly crucial to transfer the knowledge learned on the abundance of high-resource language data for low-resource speech recognition. Existing approaches simply cascade pre-trained acoustic and language models to learn the transfer from speech to text. However, how to solve the representation discrepancy of speech and text is unexplored, which hinders the utilization of acoustic and linguistic information. Moreover, previous works simply replace the embedding layer of the pre-trained language model with the acoustic features, which may cause the catastrophic forgetting problem. In this work, we introduce Wav-BERT, a cooperative acoustic and linguistic representation learning method to fuse and utilize the contextual information of speech and text. Specifically, we unify a pre-trained acoustic model (wav2vec 2.0) and a language model (BERT) into an end-to-end trainable framework. A Representation Aggregation Module is designed to aggregate acoustic and linguistic representation, and an Embedding Attention Module is introduced to incorporate acoustic information into BERT, which can effectively facilitate the cooperation of two pre-trained models and thus boost the representation learning. Extensive experiments show that our Wav-BERT significantly outperforms the existing approaches and achieves state-of-the-art performance on low-resource speech recognition.

Discipline

Graphics and Human Computer Interfaces | Programming Languages and Compilers

Research Areas

Intelligent Systems and Optimization

Areas of Excellence

Digital transformation

Publication

Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, Virtual Conference, November 7-11

First Page

2765

Last Page

2777

ISBN

9781955917094

Identifier

10.18653/V1/2021.FINDINGS-EMNLP.236

Publisher

ACL

City or Country

Punta Cana

Additional URL

https://doi.org/10.18653/V1/2021.FINDINGS-EMNLP.236

Share

COinS