MVLLaVA: An Intelligent Agent for Unified and Flexible Novel View Synthesis
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2024
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866909311398051840 |
|---|---|
| author | Jiang, Hanyu Xue, Jian Lan, Xing Hu, Guohong Lu, Ke |
| author_facet | Jiang, Hanyu Xue, Jian Lan, Xing Hu, Guohong Lu, Ke |
| contents | This paper introduces MVLLaVA, an intelligent agent designed for novel view synthesis tasks. MVLLaVA integrates multiple multi-view diffusion models with a large multimodal model, LLaVA, enabling it to handle a wide range of tasks efficiently. MVLLaVA represents a versatile and unified platform that adapts to diverse input types, including a single image, a descriptive caption, or a specific change in viewing azimuth, guided by language instructions for viewpoint generation. We carefully craft task-specific instruction templates, which are subsequently used to fine-tune LLaVA. As a result, MVLLaVA acquires the capability to generate novel view images based on user instructions, demonstrating its flexibility across diverse tasks. Experiments are conducted to validate the effectiveness of MVLLaVA, demonstrating its robust performance and versatility in tackling diverse novel view synthesis challenges. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2409_07129 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | MVLLaVA: An Intelligent Agent for Unified and Flexible Novel View Synthesis Jiang, Hanyu Xue, Jian Lan, Xing Hu, Guohong Lu, Ke Computer Vision and Pattern Recognition This paper introduces MVLLaVA, an intelligent agent designed for novel view synthesis tasks. MVLLaVA integrates multiple multi-view diffusion models with a large multimodal model, LLaVA, enabling it to handle a wide range of tasks efficiently. MVLLaVA represents a versatile and unified platform that adapts to diverse input types, including a single image, a descriptive caption, or a specific change in viewing azimuth, guided by language instructions for viewpoint generation. We carefully craft task-specific instruction templates, which are subsequently used to fine-tune LLaVA. As a result, MVLLaVA acquires the capability to generate novel view images based on user instructions, demonstrating its flexibility across diverse tasks. Experiments are conducted to validate the effectiveness of MVLLaVA, demonstrating its robust performance and versatility in tackling diverse novel view synthesis challenges. |
| title | MVLLaVA: An Intelligent Agent for Unified and Flexible Novel View Synthesis |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2409.07129 |