ClipTTT: CLIP-Guided Test-Time Training Helps LVLMs See Better
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2026
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866908917071609856 |
|---|---|
| author | Nath, Mriganka Das, Anurag Xie, Jiahao Schiele, Bernt |
| author_facet | Nath, Mriganka Das, Anurag Xie, Jiahao Schiele, Bernt |
| contents | Large vision-language models (LVLMs) tend to hallucinate, especially when visual inputs are corrupted at test time. We show that such corruptions act as additional distribution shifts, significantly amplifying hallucination rates in real-world applications. To address this, we propose CLIP-guided Test-Time Training (ClipTTT), a method to adapt LVLMs under degraded conditions on the fly with a single test sample. Specifically, we leverage the image-text alignment strength of a pre-trained CLIP model as a stable guidance signal to identify reliable self-supervision targets, enabling rapid adaptation without altering the base LVLMs. Extensive experiments on standard hallucination benchmarks, with 15 common corruptions, demonstrate that ClipTTT effectively mitigates hallucinations and improves descriptive faithfulness under visual corruptions. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2603_26486 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | ClipTTT: CLIP-Guided Test-Time Training Helps LVLMs See Better Nath, Mriganka Das, Anurag Xie, Jiahao Schiele, Bernt Computer Vision and Pattern Recognition Large vision-language models (LVLMs) tend to hallucinate, especially when visual inputs are corrupted at test time. We show that such corruptions act as additional distribution shifts, significantly amplifying hallucination rates in real-world applications. To address this, we propose CLIP-guided Test-Time Training (ClipTTT), a method to adapt LVLMs under degraded conditions on the fly with a single test sample. Specifically, we leverage the image-text alignment strength of a pre-trained CLIP model as a stable guidance signal to identify reliable self-supervision targets, enabling rapid adaptation without altering the base LVLMs. Extensive experiments on standard hallucination benchmarks, with 15 common corruptions, demonstrate that ClipTTT effectively mitigates hallucinations and improves descriptive faithfulness under visual corruptions. |
| title | ClipTTT: CLIP-Guided Test-Time Training Helps LVLMs See Better |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2603.26486 |