Identity-aware variational autoencoder for face swapping
Publication Type
Journal Article
Publication Date
1-2024
Abstract
Face swapping aims to transfer the identity of a source face to a target face image while preserving the target attributes (e.g., facial expression, head pose, illumination, and background). Most existing methods use a face recognition model to extract global features from the source face and directly fuse them with the target to generate a swapping result. However, identity-irrelevant attributes (e.g., hairstyle and facial appearances) contribute a lot to the recognition task, and thus swapping this task-specific feature inevitably interfuses source attributes with target ones. In this paper, we propose an identity-aware variational autoencoder (ID-VAE) based face swapping framework, dubbed VAFSwap, which learns disentangled identity and attribute representations for high-fidelity face swapping. In particular, we overcome the unpaired training barrier of VAE and impose a proxy identity on the latent space by exploiting the weak supervision from an auxiliary image set whose identity is averaged from multiple collected face images. To explicitly guide the identity fusion, we further devise an identity-associated matrix that corresponds different face regions with their identity representations to perform identity-related feature interactions. Finally, we incorporate spatial dimensions into the latent space and exploit the generative priors of a pre-trained face generator, allowing the effective elimination of noticeable swapping artifacts. Extensive experiments on the FaceForensics++ and CelebA-HQ datasets demonstrate that our method outperforms the state-of-the-art significantly.
Keywords
Decoding, Face recognition, Face swapping, Faces, Shape, Task analysis, Three-dimensional displays, Training, Variational autoencoder, Weak-supervised training
Discipline
Graphics and Human Computer Interfaces | Software Engineering
Research Areas
Software and Cyber-Physical Systems
Publication
IEEE Transactions on Circuits and Systems for Video Technology
ISSN
1051-8215
Identifier
10.1109/TCSVT.2024.3349909
Publisher
Institute of Electrical and Electronics Engineers
Citation
LI, Zonglin; ZHANG, Zhaoxin; HE, Shengfeng; MENG, Quanling; ZHANG, Shengping; ZHONG, Bineng; and JI, Rongrong.
Identity-aware variational autoencoder for face swapping. (2024). IEEE Transactions on Circuits and Systems for Video Technology.
Available at: https://ink.library.smu.edu.sg/sis_research/8637
Additional URL
https://doi.org/10.1109/TCSVT.2024.3349909