Salvato in:
| Autori principali: | , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2025
|
| Soggetti: | |
| Accesso online: | https://arxiv.org/abs/2507.00868 |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866911033085394944 |
|---|---|
| author | Reiß, Simon Marinov, Zdravko Jaus, Alexander Seibold, Constantin Sarfraz, M. Saquib Rodner, Erik Stiefelhagen, Rainer |
| author_facet | Reiß, Simon Marinov, Zdravko Jaus, Alexander Seibold, Constantin Sarfraz, M. Saquib Rodner, Erik Stiefelhagen, Rainer |
| contents | In this paper, we explore the potential of visual in-context learning to enable a single model to handle multiple tasks and adapt to new tasks during test time without re-training. Unlike previous approaches, our focus is on training in-context learners to adapt to sequences of tasks, rather than individual tasks. Our goal is to solve complex tasks that involve multiple intermediate steps using a single model, allowing users to define entire vision pipelines flexibly at test time. To achieve this, we first examine the properties and limitations of visual in-context learning architectures, with a particular focus on the role of codebooks. We then introduce a novel method for training in-context learners using a synthetic compositional task generation engine. This engine bootstraps task sequences from arbitrary segmentation datasets, enabling the training of visual in-context learners for compositional tasks. Additionally, we investigate different masking-based training objectives to gather insights into how to train models better for solving complex, compositional tasks. Our exploration not only provides important insights especially for multi-modal medical task sequences but also highlights challenges that need to be addressed. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2507_00868 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Is Visual in-Context Learning for Compositional Medical Tasks within Reach? Reiß, Simon Marinov, Zdravko Jaus, Alexander Seibold, Constantin Sarfraz, M. Saquib Rodner, Erik Stiefelhagen, Rainer Computer Vision and Pattern Recognition In this paper, we explore the potential of visual in-context learning to enable a single model to handle multiple tasks and adapt to new tasks during test time without re-training. Unlike previous approaches, our focus is on training in-context learners to adapt to sequences of tasks, rather than individual tasks. Our goal is to solve complex tasks that involve multiple intermediate steps using a single model, allowing users to define entire vision pipelines flexibly at test time. To achieve this, we first examine the properties and limitations of visual in-context learning architectures, with a particular focus on the role of codebooks. We then introduce a novel method for training in-context learners using a synthetic compositional task generation engine. This engine bootstraps task sequences from arbitrary segmentation datasets, enabling the training of visual in-context learners for compositional tasks. Additionally, we investigate different masking-based training objectives to gather insights into how to train models better for solving complex, compositional tasks. Our exploration not only provides important insights especially for multi-modal medical task sequences but also highlights challenges that need to be addressed. |
| title | Is Visual in-Context Learning for Compositional Medical Tasks within Reach? |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2507.00868 |