Action100M: A Large-scale Video Action Dataset
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2026
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866912826490093568 |
|---|---|
| author | Chen, Delong Kasarla, Tejaswi Bang, Yejin Shukor, Mustafa Chung, Willy Yu, Jade Bolourchi, Allen Moutakanni, Theo Fung, Pascale |
| author_facet | Chen, Delong Kasarla, Tejaswi Bang, Yejin Shukor, Mustafa Chung, Willy Yu, Jade Bolourchi, Allen Moutakanni, Theo Fung, Pascale |
| contents | Inferring physical actions from visual observations is a fundamental capability for advancing machine intelligence in the physical world. Achieving this requires large-scale, open-vocabulary video action datasets that span broad domains. We introduce Action100M, a large-scale dataset constructed from 1.2M Internet instructional videos (14.6 years of duration), yielding O(100 million) temporally localized segments with open-vocabulary action supervision and rich captions. Action100M is generated by a fully automated pipeline that (i) performs hierarchical temporal segmentation using V-JEPA 2 embeddings, (ii) produces multi-level frame and segment captions organized as a Tree-of-Captions, and (iii) aggregates evidence with a reasoning model (GPT-OSS-120B) under a multi-round Self-Refine procedure to output structured annotations (brief/detailed action, actor, brief/detailed caption). Training VL-JEPA on Action100M demonstrates consistent data-scaling improvements and strong zero-shot performance across diverse action recognition benchmarks, establishing Action100M as a new foundation for scalable research in video understanding and world modeling. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2601_10592 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | Action100M: A Large-scale Video Action Dataset Chen, Delong Kasarla, Tejaswi Bang, Yejin Shukor, Mustafa Chung, Willy Yu, Jade Bolourchi, Allen Moutakanni, Theo Fung, Pascale Computer Vision and Pattern Recognition Inferring physical actions from visual observations is a fundamental capability for advancing machine intelligence in the physical world. Achieving this requires large-scale, open-vocabulary video action datasets that span broad domains. We introduce Action100M, a large-scale dataset constructed from 1.2M Internet instructional videos (14.6 years of duration), yielding O(100 million) temporally localized segments with open-vocabulary action supervision and rich captions. Action100M is generated by a fully automated pipeline that (i) performs hierarchical temporal segmentation using V-JEPA 2 embeddings, (ii) produces multi-level frame and segment captions organized as a Tree-of-Captions, and (iii) aggregates evidence with a reasoning model (GPT-OSS-120B) under a multi-round Self-Refine procedure to output structured annotations (brief/detailed action, actor, brief/detailed caption). Training VL-JEPA on Action100M demonstrates consistent data-scaling improvements and strong zero-shot performance across diverse action recognition benchmarks, establishing Action100M as a new foundation for scalable research in video understanding and world modeling. |
| title | Action100M: A Large-scale Video Action Dataset |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2601.10592 |