Do pre-trained models benefit knowledge graph completion? A reliable evaluation and a reasonable approach

Author

Xin LV
Yankai LIN
Yixin CAO, Singapore Management UniversityFollow
Lei HOU
Juanzi LI
Zhiyuan LIU
Peng LI
Jie ZHOU

Publication Type

Conference Proceeding Article

Version

publishedVersion

Publication Date

5-2022

Abstract

In recent years, pre-trained language models (PLMs) have been shown to capture factual knowledge from massive texts, which encourages the proposal of PLM-based knowledge graph completion (KGC) models. However, these models are still quite behind the SOTA KGC models in terms of performance. In this work, we find two main reasons for the weak performance: (1) Inaccurate evaluation setting. The evaluation setting under the closed-world assumption (CWA) may underestimate the PLM-based KGC models since they introduce more external knowledge; (2) Inappropriate utilization of PLMs. Most PLM-based KGC models simply splice the labels of entities and relations as inputs, leading to incoherent sentences that do not take full advantage of the implicit knowledge in PLMs. To alleviate these problems, we highlight a more accurate evaluation setting under the open-world assumption (OWA), which manual checks the correctness of knowledge that is not in KGs. Moreover, motivated by prompt tuning, we propose a novel PLM-based KGC model named PKGC. The basic idea is to convert each triple and its support information into natural prompt sentences, which is further fed into PLMs for classification. Experiment results on two KGC datasets demonstrate OWA is more reliable for evaluating KGC, especially on the link prediction, and the effectiveness of our PKCG model on both CWA and OWA settings.

Discipline

Databases and Information Systems | Graphics and Human Computer Interfaces

Research Areas

Data Science and Engineering

Publication

Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics: System Demonstrations, Dublin, Ireland, 2022 May 22-27

First Page

3570

Last Page

3581

Identifier

10.18653/v1/2022.findings-acl.282

Publisher

Association for Computational Linguistics

City or Country

Dublin, Ireland

Citation

LV, Xin; LIN, Yankai; CAO, Yixin; HOU, Lei; LI, Juanzi; LIU, Zhiyuan; LI, Peng; and ZHOU, Jie. Do pre-trained models benefit knowledge graph completion? A reliable evaluation and a reasonable approach. (2022). Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics: System Demonstrations, Dublin, Ireland, 2022 May 22-27. 3570-3581.
Available at: https://ink.library.smu.edu.sg/sis_research/7446

Creative Commons License

This work is licensed under a Creative Commons Attribution-NonCommercial-No Derivative Works 4.0 International License.

Additional URL

http://doi.org/10.18653/v1/2022.findings-acl.282