TAR3D: Creating High-Quality 3D Assets via Next-Part Prediction
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866911099750711296 |
|---|---|
| author | Zhang, Xuying Liu, Yutong Li, Yangguang Zhang, Renrui Liu, Yufei Wang, Kai Ouyang, Wanli Xiong, Zhiwei Gao, Peng Hou, Qibin Cheng, Ming-Ming |
| author_facet | Zhang, Xuying Liu, Yutong Li, Yangguang Zhang, Renrui Liu, Yufei Wang, Kai Ouyang, Wanli Xiong, Zhiwei Gao, Peng Hou, Qibin Cheng, Ming-Ming |
| contents | We present TAR3D, a novel framework that consists of a 3D-aware Vector Quantized-Variational AutoEncoder (VQ-VAE) and a Generative Pre-trained Transformer (GPT) to generate high-quality 3D assets. The core insight of this work is to migrate the multimodal unification and promising learning capabilities of the next-token prediction paradigm to conditional 3D object generation. To achieve this, the 3D VQ-VAE first encodes a wide range of 3D shapes into a compact triplane latent space and utilizes a set of discrete representations from a trainable codebook to reconstruct fine-grained geometries under the supervision of query point occupancy. Then, the 3D GPT, equipped with a custom triplane position embedding called TriPE, predicts the codebook index sequence with prefilling prompt tokens in an autoregressive manner so that the composition of 3D geometries can be modeled part by part. Extensive experiments on ShapeNet and Objaverse demonstrate that TAR3D can achieve superior generation quality over existing methods in text-to-3D and image-to-3D tasks |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2412_16919 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | TAR3D: Creating High-Quality 3D Assets via Next-Part Prediction Zhang, Xuying Liu, Yutong Li, Yangguang Zhang, Renrui Liu, Yufei Wang, Kai Ouyang, Wanli Xiong, Zhiwei Gao, Peng Hou, Qibin Cheng, Ming-Ming Computer Vision and Pattern Recognition We present TAR3D, a novel framework that consists of a 3D-aware Vector Quantized-Variational AutoEncoder (VQ-VAE) and a Generative Pre-trained Transformer (GPT) to generate high-quality 3D assets. The core insight of this work is to migrate the multimodal unification and promising learning capabilities of the next-token prediction paradigm to conditional 3D object generation. To achieve this, the 3D VQ-VAE first encodes a wide range of 3D shapes into a compact triplane latent space and utilizes a set of discrete representations from a trainable codebook to reconstruct fine-grained geometries under the supervision of query point occupancy. Then, the 3D GPT, equipped with a custom triplane position embedding called TriPE, predicts the codebook index sequence with prefilling prompt tokens in an autoregressive manner so that the composition of 3D geometries can be modeled part by part. Extensive experiments on ShapeNet and Objaverse demonstrate that TAR3D can achieve superior generation quality over existing methods in text-to-3D and image-to-3D tasks |
| title | TAR3D: Creating High-Quality 3D Assets via Next-Part Prediction |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2412.16919 |