CounterVid: Counterfactual Video Generation for Mitigating Action and Temporal Hallucinations in Video-Language Models
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2026
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866912809877504000 |
|---|---|
| author | Poppi, Tobia Uzkent, Burak Garg, Amanmeet Porto, Lucas Kessler, Garin Yang, Yezhou Cornia, Marcella Baraldi, Lorenzo Cucchiara, Rita Schiffers, Florian |
| author_facet | Poppi, Tobia Uzkent, Burak Garg, Amanmeet Porto, Lucas Kessler, Garin Yang, Yezhou Cornia, Marcella Baraldi, Lorenzo Cucchiara, Rita Schiffers, Florian |
| contents | Video-language models (VLMs) achieve strong multimodal understanding but remain prone to hallucinations, especially when reasoning about actions and temporal order. Existing mitigation strategies, such as textual filtering or random video perturbations, often fail to address the root cause: over-reliance on language priors rather than fine-grained visual dynamics. We propose a scalable framework for counterfactual video generation that synthesizes videos differing only in actions or temporal structure while preserving scene context. Our pipeline combines multimodal LLMs for action proposal and editing guidance with diffusion-based image and video models to generate semantic hard negatives at scale. Using this framework, we build CounterVid, a synthetic dataset of ~26k preference pairs targeting action recognition and temporal reasoning. We further introduce MixDPO, a unified Direct Preference Optimization approach that jointly leverages textual and visual preferences. Fine-tuning Qwen2.5-VL with MixDPO yields consistent improvements, notably in temporal ordering, and transfers effectively to standard video hallucination benchmarks. Code and models will be made publicly available. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2601_04778 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | CounterVid: Counterfactual Video Generation for Mitigating Action and Temporal Hallucinations in Video-Language Models Poppi, Tobia Uzkent, Burak Garg, Amanmeet Porto, Lucas Kessler, Garin Yang, Yezhou Cornia, Marcella Baraldi, Lorenzo Cucchiara, Rita Schiffers, Florian Computer Vision and Pattern Recognition Artificial Intelligence Computation and Language Multimedia Video-language models (VLMs) achieve strong multimodal understanding but remain prone to hallucinations, especially when reasoning about actions and temporal order. Existing mitigation strategies, such as textual filtering or random video perturbations, often fail to address the root cause: over-reliance on language priors rather than fine-grained visual dynamics. We propose a scalable framework for counterfactual video generation that synthesizes videos differing only in actions or temporal structure while preserving scene context. Our pipeline combines multimodal LLMs for action proposal and editing guidance with diffusion-based image and video models to generate semantic hard negatives at scale. Using this framework, we build CounterVid, a synthetic dataset of ~26k preference pairs targeting action recognition and temporal reasoning. We further introduce MixDPO, a unified Direct Preference Optimization approach that jointly leverages textual and visual preferences. Fine-tuning Qwen2.5-VL with MixDPO yields consistent improvements, notably in temporal ordering, and transfers effectively to standard video hallucination benchmarks. Code and models will be made publicly available. |
| title | CounterVid: Counterfactual Video Generation for Mitigating Action and Temporal Hallucinations in Video-Language Models |
| topic | Computer Vision and Pattern Recognition Artificial Intelligence Computation and Language Multimedia |
| url | https://arxiv.org/abs/2601.04778 |