Publication Type
Conference Proceeding Article
Version
acceptedVersion
Publication Date
11-2024
Abstract
Despite having tremendous progress in image-to-3D generation, existing methods still struggle to produce multi-view consistent images with high-resolution textures in detail, especially in the paradigm of 2D diffusion that lacks 3D awareness. In this work, we present High-resolution Image-to-3D model (Hi3D), a new video diffusion based paradigm that redefines a single image to multi-view images as 3D-aware sequential image generation (i.e., orbital video generation). This methodology delves into the underlying temporal consistency knowledge in video diffusion model that generalizes well to geometry consistency across multiple views in 3D generation. Technically, Hi3D first empowers the pre-trained video diffusion model with 3D-aware prior (camera pose condition), yielding multi-view images with low-resolution texture details. A 3D-aware video-to-video refiner is learnt to further scale up the multi-view images with high-resolution texture details. Such high-resolution multi-view images are further augmented with novel views through 3D Gaussian Splatting, which are finally leveraged to obtain high-fidelity meshes via 3D reconstruction. Extensive experiments on both novel view synthesis and single view reconstruction demonstrate that our Hi3D manages to produce superior multi-view consistency images with highly-detailed textures. Source code and data are available at https://github.com/yanghb22-fdu/Hi3D-Official.
Keywords
high resolution, image-to-3d generation, video diffusion model
Discipline
Graphics and Human Computer Interfaces
Research Areas
Intelligent Systems and Optimization
Areas of Excellence
Digital transformation
Publication
MM '24: Proceedings of the 32nd ACM International Conference on Multimedia, Melbourne, Australia, October 28-November 1
First Page
6870
Last Page
6879
ISBN
9798400706868
Identifier
10.1145/3664647.3681634
Publisher
ACM
City or Country
New York
Citation
YANG, Haibo; CHEN, Yang; PAN, Yingwei; YAO, Ting; CHEN, Zhineng; NGO, Chong-wah; and MEI, Tao.
Hi3D: Pursuing high-resolution image-to-3D generation with video diffusion models. (2024). MM '24: Proceedings of the 32nd ACM International Conference on Multimedia, Melbourne, Australia, October 28-November 1. 6870-6879.
Available at: https://ink.library.smu.edu.sg/sis_research/9871
Creative Commons License
This work is licensed under a Creative Commons Attribution-NonCommercial-No Derivative Works 4.0 International License.
Additional URL
https://doi.org/10.1145/3664647.3681634
