Publication Type

Conference Proceeding Article

Version

acceptedVersion

Publication Date

10-2023

Abstract

GAN inversion is indispensable for applying the powerful editability of GAN to real images. However, existing methods invert video frames individually often leading to undesired inconsistent results over time. In this paper, we propose a unified recurrent framework, named Recurrent vIdeo GAN Inversion and eDiting (RIGID), to explicitly and simultaneously enforce temporally coherent GAN inversion and facial editing of real videos. Our approach models the temporal relations between current and previous frames from three aspects. To enable a faithful real video reconstruction, we first maximize the inversion fidelity and consistency by learning a temporal compensated latent code. Second, we observe incoherent noises lie in the high-frequency domain that can be disentangled from the latent space. Third, to remove the inconsistency after attribute manipulation, we propose an in-between frame composition constraint such that the arbitrary frame must be a direct composite of its neighboring frames. Our unified framework learns the inherent coherence between input frames in an end-to-end manner, and therefore it is agnostic to a specific attribute and can be applied to arbitrary editing of the same video without re-training. Extensive experiments demonstrate that RIGID outperforms state-of-the-art methods qualitatively and quantitatively in both inversion and editing tasks. The deliverables can be found in https://cnnlstm.github.io/RIGID.

Discipline

Computer Sciences | Graphics and Human Computer Interfaces

Research Areas

Software and Cyber-Physical Systems

Publication

2023 IEEE/CVF International Conference on Computer Vision (ICCV): Paris, October 2-6: Proceedings

First Page

13645

Last Page

13655

ISBN

9798350307184

Identifier

10.1109/ICCV51070.2023.01259

Publisher

IEEE Computer Society

City or Country

Washington, DC

Copyright Owner and License

Authors

Additional URL

https://doi.org/10.1109/ICCV51070.2023.01259

Share

COinS