Identity-aware variational autoencoder for face swapping

Publication Type

Journal Article

Publication Date

1-2024

Abstract

Face swapping aims to transfer the identity of a source face to a target face image while preserving the target attributes (e.g., facial expression, head pose, illumination, and background). Most existing methods use a face recognition model to extract global features from the source face and directly fuse them with the target to generate a swapping result. However, identity-irrelevant attributes (e.g., hairstyle and facial appearances) contribute a lot to the recognition task, and thus swapping this task-specific feature inevitably interfuses source attributes with target ones. In this paper, we propose an identity-aware variational autoencoder (ID-VAE) based face swapping framework, dubbed VAFSwap, which learns disentangled identity and attribute representations for high-fidelity face swapping. In particular, we overcome the unpaired training barrier of VAE and impose a proxy identity on the latent space by exploiting the weak supervision from an auxiliary image set whose identity is averaged from multiple collected face images. To explicitly guide the identity fusion, we further devise an identity-associated matrix that corresponds different face regions with their identity representations to perform identity-related feature interactions. Finally, we incorporate spatial dimensions into the latent space and exploit the generative priors of a pre-trained face generator, allowing the effective elimination of noticeable swapping artifacts. Extensive experiments on the FaceForensics++ and CelebA-HQ datasets demonstrate that our method outperforms the state-of-the-art significantly.

Keywords

Decoding, Face recognition, Face swapping, Faces, Shape, Task analysis, Three-dimensional displays, Training, Variational autoencoder, Weak-supervised training

Discipline

Graphics and Human Computer Interfaces | Software Engineering

Research Areas

Software and Cyber-Physical Systems

Publication

IEEE Transactions on Circuits and Systems for Video Technology

ISSN

1051-8215

Identifier

10.1109/TCSVT.2024.3349909

Publisher

Institute of Electrical and Electronics Engineers

Additional URL

https://doi.org/10.1109/TCSVT.2024.3349909

This document is currently not available here.

Share

COinS