Publication Type
Conference Proceeding Article
Version
submittedVersion
Publication Date
12-2024
Abstract
The swift advancement in Multimodal LLMs (MLLMs) also presents significant challenges for effective knowledge editing. Current methods, including intrinsic knowledge editing and external knowledge resorting, each possess strengths and weaknesses, struggling to balance the desired properties of reliability, generality, and locality when applied to MLLMs. In this paper, we propose UniKE, a novel multimodal editing method that establishes a unified perspective and paradigm for intrinsic knowledge editing and external knowledge resorting. Both types of knowledge are conceptualized as vectorized key-value memories, with the corresponding editing processes resembling the assimilation and accommodation phases of human cognition, conducted at the same semantic levels. Within such a unified framework, we further promote knowledge collaboration by disentangling the knowledge representations into the semantic and truthfulness spaces.
Keywords
Multimodal LLMs, Knowledge editing, Intrinsic knowledge editing, External knowledge resorting
Discipline
Databases and Information Systems
Research Areas
Data Science and Engineering
Publication
Proceedings of the 38th Conference on Neural Information Processing Systems, Vancouver, Canada, 2024 December 10-15
First Page
1
Last Page
18
Publisher
The Neural Information Processing Systems Foundation
City or Country
California
Citation
PAN, Kaihang; FAN, Zhaoyu; LI, Juncheng; YU, Qifan; FEI, Hao; TANG, Siliang; HONG, Richang; ZHANG, Hanwang; and Qianru SUN.
Towards unified multimodal editing with enhanced knowledge collaboration. (2024). Proceedings of the 38th Conference on Neural Information Processing Systems, Vancouver, Canada, 2024 December 10-15. 1-18.
Available at: https://ink.library.smu.edu.sg/sis_research/9401
Creative Commons License
This work is licensed under a Creative Commons Attribution-NonCommercial-No Derivative Works 4.0 International License.