Pro-pulse: Learning progressive encoders of latent semantics in gans for photo upsampling
Publication Type
Journal Article
Publication Date
1-2022
Abstract
The state-of-the-art photo upsampling method, PULSE, demonstrates that a sharp, high-resolution (HR) version of a given low-resolution (LR) input can be obtained by exploring the latent space of generative models. However, mapping an extreme LR input (16(2)) directly to an HR image (1024(2)) is too ambiguous to preserve faithful local facial semantics. In this paper, we propose an enhanced upsampling approach, Pro-PULSE, that addresses the issues of semantic inconsistency and optimization complexity. Our idea is to learn an encoder that progressively constructs the HR latent codes in the extended W+ latent space of StyleGAN. This design divides the complex 64x upsampling problem into several steps, and therefore small-scale facial semantics can be inherited from one end to the other. In particular, we train two encoders, the base encoder maps latent vectors in W space and serves as a foundation of the HR latent vector, while the second scale-specific encoder performed in W+ space gradually replaces the previous vector produced by the base encoder at each scale. This process produces intermediate side-outputs, which injects deep supervision into the training of encoder. Extensive experiments demonstrate superiorities over the latest latent space exploration methods, in terms of efficiency, quantitative quality metrics, and qualitative visual results.
Keywords
Photo upsampling, GANs, progressive learning, latent space
Discipline
Information Security
Research Areas
Information Systems and Management
Publication
IEEE Transactions on Image Processing
Volume
31
First Page
1230
Last Page
1242
ISSN
1057-7149
Identifier
10.1109/TIP.2022.3140603
Publisher
Institute of Electrical and Electronics Engineers
Citation
ZHOU, Yang; XU, Yangyang; DU, Yong; WEN, Qiang; and HE, Shengfeng.
Pro-pulse: Learning progressive encoders of latent semantics in gans for photo upsampling. (2022). IEEE Transactions on Image Processing. 31, 1230-1242.
Available at: https://ink.library.smu.edu.sg/sis_research/7877
Additional URL
https://doi.org/10.1109/TIP.2022.3140603