VP-LLM: Text-Driven 3D Volume Completion with Large Language Models through Patchification
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866909219777675264 |
|---|---|
| author | Liu, Jianmeng Liu, Yichen Zhang, Yuyao Meng, Zeyuan Tai, Yu-Wing Tang, Chi-Keung |
| author_facet | Liu, Jianmeng Liu, Yichen Zhang, Yuyao Meng, Zeyuan Tai, Yu-Wing Tang, Chi-Keung |
| contents | Recent conditional 3D completion works have mainly relied on CLIP or BERT to encode textual information, which cannot support complex instruction. Meanwhile, large language models (LLMs) have shown great potential in multi-modal understanding and generation tasks. Inspired by the recent advancements of LLM, we present Volume Patch LLM (VP-LLM), which leverages LLMs to perform conditional 3D completion in a single-forward pass. To integrate a 3D model into the LLM tokenization configuration, the incomplete 3D object is first divided into small patches that can be encoded independently. These encoded patches are then fed into an LLM along with the text prompt, instructing the LLM to capture the relations between these patches as well as injecting semantic meanings into the 3D object. Our results demonstrate a strong ability of LLMs to interpret complex text instructions and understand 3D objects, surpassing state-of-the-art diffusion-based 3D completion models in generation quality. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2406_05543 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | VP-LLM: Text-Driven 3D Volume Completion with Large Language Models through Patchification Liu, Jianmeng Liu, Yichen Zhang, Yuyao Meng, Zeyuan Tai, Yu-Wing Tang, Chi-Keung Computer Vision and Pattern Recognition Artificial Intelligence Recent conditional 3D completion works have mainly relied on CLIP or BERT to encode textual information, which cannot support complex instruction. Meanwhile, large language models (LLMs) have shown great potential in multi-modal understanding and generation tasks. Inspired by the recent advancements of LLM, we present Volume Patch LLM (VP-LLM), which leverages LLMs to perform conditional 3D completion in a single-forward pass. To integrate a 3D model into the LLM tokenization configuration, the incomplete 3D object is first divided into small patches that can be encoded independently. These encoded patches are then fed into an LLM along with the text prompt, instructing the LLM to capture the relations between these patches as well as injecting semantic meanings into the 3D object. Our results demonstrate a strong ability of LLMs to interpret complex text instructions and understand 3D objects, surpassing state-of-the-art diffusion-based 3D completion models in generation quality. |
| title | VP-LLM: Text-Driven 3D Volume Completion with Large Language Models through Patchification |
| topic | Computer Vision and Pattern Recognition Artificial Intelligence |
| url | https://arxiv.org/abs/2406.05543 |