Toward a Diffusion-Based Generalist for Dense Vision Tasks
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866909234756583424 |
|---|---|
| author | Fan, Yue Xian, Yongqin Zhai, Xiaohua Kolesnikov, Alexander Naeem, Muhammad Ferjad Schiele, Bernt Tombari, Federico |
| author_facet | Fan, Yue Xian, Yongqin Zhai, Xiaohua Kolesnikov, Alexander Naeem, Muhammad Ferjad Schiele, Bernt Tombari, Federico |
| contents | Building generalized models that can solve many computer vision tasks simultaneously is an intriguing direction. Recent works have shown image itself can be used as a natural interface for general-purpose visual perception and demonstrated inspiring results. In this paper, we explore diffusion-based vision generalists, where we unify different types of dense prediction tasks as conditional image generation and re-purpose pre-trained diffusion models for it. However, directly applying off-the-shelf latent diffusion models leads to a quantization issue. Thus, we propose to perform diffusion in pixel space and provide a recipe for finetuning pre-trained text-to-image diffusion models for dense vision tasks. In experiments, we evaluate our method on four different types of tasks and show competitive performance to the other vision generalists. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2407_00503 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Toward a Diffusion-Based Generalist for Dense Vision Tasks Fan, Yue Xian, Yongqin Zhai, Xiaohua Kolesnikov, Alexander Naeem, Muhammad Ferjad Schiele, Bernt Tombari, Federico Computer Vision and Pattern Recognition Building generalized models that can solve many computer vision tasks simultaneously is an intriguing direction. Recent works have shown image itself can be used as a natural interface for general-purpose visual perception and demonstrated inspiring results. In this paper, we explore diffusion-based vision generalists, where we unify different types of dense prediction tasks as conditional image generation and re-purpose pre-trained diffusion models for it. However, directly applying off-the-shelf latent diffusion models leads to a quantization issue. Thus, we propose to perform diffusion in pixel space and provide a recipe for finetuning pre-trained text-to-image diffusion models for dense vision tasks. In experiments, we evaluate our method on four different types of tasks and show competitive performance to the other vision generalists. |
| title | Toward a Diffusion-Based Generalist for Dense Vision Tasks |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2407.00503 |