LAP: A Language-Aware Planning Model For Procedure Planning In Instructional Videos
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2026
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866915851188305920 |
|---|---|
| author | Shi, Lei Aregbede, Victor Persson, Andreas Längkvist, Martin Loutfi, Amy Lowry, Stephanie |
| author_facet | Shi, Lei Aregbede, Victor Persson, Andreas Längkvist, Martin Loutfi, Amy Lowry, Stephanie |
| contents | Procedure planning requires a model to predict a sequence of actions that transform a start visual observation into a goal in instructional videos. While most existing methods rely primarily on visual observations as input, they often struggle with the inherent ambiguity where different actions can appear visually similar. In this work, we argue that language descriptions offer a more distinctive representation in the latent space for procedure planning. We introduce Language-Aware Planning (LAP), a novel method that leverages the expressiveness of language to bridge visual observation and planning. LAP uses a finetuned Vision Language Model (VLM) to translate visual observations into text descriptions and to predict actions and extract text embeddings. These text embeddings are more distinctive than visual embeddings and are used in a diffusion model for planning action sequences. We evaluate LAP on three procedure planning benchmarks: CrossTask, Coin, and NIV. LAP achieves new state-of-the-art performance across multiple metrics and time horizons by large margin, demonstrating the significant advantage of language-aware planning. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2603_09743 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | LAP: A Language-Aware Planning Model For Procedure Planning In Instructional Videos Shi, Lei Aregbede, Victor Persson, Andreas Längkvist, Martin Loutfi, Amy Lowry, Stephanie Computer Vision and Pattern Recognition Procedure planning requires a model to predict a sequence of actions that transform a start visual observation into a goal in instructional videos. While most existing methods rely primarily on visual observations as input, they often struggle with the inherent ambiguity where different actions can appear visually similar. In this work, we argue that language descriptions offer a more distinctive representation in the latent space for procedure planning. We introduce Language-Aware Planning (LAP), a novel method that leverages the expressiveness of language to bridge visual observation and planning. LAP uses a finetuned Vision Language Model (VLM) to translate visual observations into text descriptions and to predict actions and extract text embeddings. These text embeddings are more distinctive than visual embeddings and are used in a diffusion model for planning action sequences. We evaluate LAP on three procedure planning benchmarks: CrossTask, Coin, and NIV. LAP achieves new state-of-the-art performance across multiple metrics and time horizons by large margin, demonstrating the significant advantage of language-aware planning. |
| title | LAP: A Language-Aware Planning Model For Procedure Planning In Instructional Videos |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2603.09743 |