From Image to Video: An Empirical Study of Diffusion Representations
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2025
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866908275821248512 |
|---|---|
| author | Vélez, Pedro Polanía, Luisa F. Yang, Yi Zhang, Chuhan Kabra, Rishabh Arnab, Anurag Sajjadi, Mehdi S. M. |
| author_facet | Vélez, Pedro Polanía, Luisa F. Yang, Yi Zhang, Chuhan Kabra, Rishabh Arnab, Anurag Sajjadi, Mehdi S. M. |
| contents | Diffusion models have revolutionized generative modeling, enabling unprecedented realism in image and video synthesis. This success has sparked interest in leveraging their representations for visual understanding tasks. While recent works have explored this potential for image generation, the visual understanding capabilities of video diffusion models remain largely uncharted. To address this gap, we systematically compare the same model architecture trained for video versus image generation, analyzing the performance of their latent representations on various downstream tasks including image classification, action recognition, depth estimation, and tracking. Results show that video diffusion models consistently outperform their image counterparts, though we find a striking range in the extent of this superiority. We further analyze features extracted from different layers and with varying noise levels, as well as the effect of model size and training budget on representation and generation quality. This work marks the first direct comparison of video and image diffusion objectives for visual understanding, offering insights into the role of temporal information in representation learning. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2502_07001 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | From Image to Video: An Empirical Study of Diffusion Representations Vélez, Pedro Polanía, Luisa F. Yang, Yi Zhang, Chuhan Kabra, Rishabh Arnab, Anurag Sajjadi, Mehdi S. M. Computer Vision and Pattern Recognition Artificial Intelligence Machine Learning Diffusion models have revolutionized generative modeling, enabling unprecedented realism in image and video synthesis. This success has sparked interest in leveraging their representations for visual understanding tasks. While recent works have explored this potential for image generation, the visual understanding capabilities of video diffusion models remain largely uncharted. To address this gap, we systematically compare the same model architecture trained for video versus image generation, analyzing the performance of their latent representations on various downstream tasks including image classification, action recognition, depth estimation, and tracking. Results show that video diffusion models consistently outperform their image counterparts, though we find a striking range in the extent of this superiority. We further analyze features extracted from different layers and with varying noise levels, as well as the effect of model size and training budget on representation and generation quality. This work marks the first direct comparison of video and image diffusion objectives for visual understanding, offering insights into the role of temporal information in representation learning. |
| title | From Image to Video: An Empirical Study of Diffusion Representations |
| topic | Computer Vision and Pattern Recognition Artificial Intelligence Machine Learning |
| url | https://arxiv.org/abs/2502.07001 |