Publication Type
Conference Proceeding Article
Version
acceptedVersion
Publication Date
4-2021
Abstract
Pretrained using large amount of data, autoregressive language models are able to generate high quality sequences. However, these models do not perform well under hard lexical constraints as they lack fine control of content generation process. Progressive insertion-based transformers can overcome the above limitation and efficiently generate a sequence in parallel given some input tokens as constraint. These transformers however may fail to support hard lexical constraints as their generation process is more likely to terminate prematurely. The paper analyses such early termination problems and proposes the ENtity-CONstrained insertion TransformER (ENCONTER), a new insertion transformer that addresses the above pitfall without compromising much generation efficiency. We introduce a new training strategy that considers predefined hard lexical constraints (e.g., entities to be included in the generated sequence). Our experiments show that ENCONTER outperforms other baseline models in several performance metrics rendering it more suitable in practical applications.
Discipline
Databases and Information Systems | Numerical Analysis and Scientific Computing
Research Areas
Data Science and Engineering
Publication
EACL 2021: Proceedings of the 16th European Chapter of the Association for Computational Linguistics, April 19-23, Kyiv, Ukraine and Virtual
First Page
1
Last Page
10
Publisher
ACL
City or Country
Stroudsburg, PA
Embargo Period
4-15-2021
Citation
HSIEH, Lee Hsun; LEE, Yang Yin; and LIM, Ee-Peng.
ENCONTER: Entity constrained progressive sequence generation via insertion-based transformer. (2021). EACL 2021: Proceedings of the 16th European Chapter of the Association for Computational Linguistics, April 19-23, Kyiv, Ukraine and Virtual. 1-10.
Available at: https://ink.library.smu.edu.sg/sis_research/5891
Copyright Owner and License
LARC and Authors
Creative Commons License
This work is licensed under a Creative Commons Attribution-NonCommercial-No Derivative Works 4.0 International License.
Included in
Databases and Information Systems Commons, Numerical Analysis and Scientific Computing Commons
Comments
Code is available at https://github.com/LARC-CMU-SMU/Enconter