LocoMotion: Learning Motion-Focused Video-Language Representations
Fuente:
arXiv
Salvato in:
| Autori principali: | , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2024
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866913560709300224 |
|---|---|
| author | Doughty, Hazel Thoker, Fida Mohammad Snoek, Cees G. M. |
| author_facet | Doughty, Hazel Thoker, Fida Mohammad Snoek, Cees G. M. |
| contents | This paper strives for motion-focused video-language representations. Existing methods to learn video-language representations use spatial-focused data, where identifying the objects and scene is often enough to distinguish the relevant caption. We instead propose LocoMotion to learn from motion-focused captions that describe the movement and temporal progression of local object motions. We achieve this by adding synthetic motions to videos and using the parameters of these motions to generate corresponding captions. Furthermore, we propose verb-variation paraphrasing to increase the caption variety and learn the link between primitive motions and high-level verbs. With this, we are able to learn a motion-focused video-language representation. Experiments demonstrate our approach is effective for a variety of downstream tasks, particularly when limited data is available for fine-tuning. Code is available: https://hazeldoughty.github.io/Papers/LocoMotion/ |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2410_12018 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | LocoMotion: Learning Motion-Focused Video-Language Representations Doughty, Hazel Thoker, Fida Mohammad Snoek, Cees G. M. Computer Vision and Pattern Recognition Computation and Language Multimedia This paper strives for motion-focused video-language representations. Existing methods to learn video-language representations use spatial-focused data, where identifying the objects and scene is often enough to distinguish the relevant caption. We instead propose LocoMotion to learn from motion-focused captions that describe the movement and temporal progression of local object motions. We achieve this by adding synthetic motions to videos and using the parameters of these motions to generate corresponding captions. Furthermore, we propose verb-variation paraphrasing to increase the caption variety and learn the link between primitive motions and high-level verbs. With this, we are able to learn a motion-focused video-language representation. Experiments demonstrate our approach is effective for a variety of downstream tasks, particularly when limited data is available for fine-tuning. Code is available: https://hazeldoughty.github.io/Papers/LocoMotion/ |
| title | LocoMotion: Learning Motion-Focused Video-Language Representations |
| topic | Computer Vision and Pattern Recognition Computation and Language Multimedia |
| url | https://arxiv.org/abs/2410.12018 |