Saved in:
| Main Authors: | , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2026
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2603.23149 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866912980331921408 |
|---|---|
| author | Pappa, Massimiliano Romani, Luca Sacco, Valentino Palma, Alessio Lathuilière, Stéphane Galasso, Fabio Alameda-Pineda, Xavier Spinelli, Indro |
| author_facet | Pappa, Massimiliano Romani, Luca Sacco, Valentino Palma, Alessio Lathuilière, Stéphane Galasso, Fabio Alameda-Pineda, Xavier Spinelli, Indro |
| contents | Deploying safety-critical agents requires anticipating the consequences of actions before they are executed. While world models offer a paradigm for this proactive foresight, current approaches relying on visual simulation incur prohibitive latencies, often exceeding several seconds per step. In this work, we challenge the assumption that visual processing is necessary for failure prevention. We show that a trained policy's latent state, combined with its planned actions, already encodes sufficient information to anticipate action outcomes, making visual simulation redundant for failure prevention. To this end, we introduce DILLO (DIstiLLed Language-ActiOn World Model), a fast steering layer that shifts the paradigm from "simulate-then-act" to "describe-then-act." DILLO is trained via cross-modal distillation, where a privileged Vision Language Model teacher annotates offline trajectories and a latent-conditioned Large Language Model student learns to predict semantic outcomes. This creates a text-only inference path, bypassing heavy visual generation entirely, achieving a 14x speedup over baselines. Experiments on MetaWorld and LIBERO demonstrate that DILLO produces high-fidelity descriptions of the next state and is able to steer the policy, improving episode success rate by up to 15 pp and 9.3 pp on average across tasks. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2603_23149 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | Describe-Then-Act: Proactive Agent Steering via Distilled Language-Action World Models Pappa, Massimiliano Romani, Luca Sacco, Valentino Palma, Alessio Lathuilière, Stéphane Galasso, Fabio Alameda-Pineda, Xavier Spinelli, Indro Artificial Intelligence Deploying safety-critical agents requires anticipating the consequences of actions before they are executed. While world models offer a paradigm for this proactive foresight, current approaches relying on visual simulation incur prohibitive latencies, often exceeding several seconds per step. In this work, we challenge the assumption that visual processing is necessary for failure prevention. We show that a trained policy's latent state, combined with its planned actions, already encodes sufficient information to anticipate action outcomes, making visual simulation redundant for failure prevention. To this end, we introduce DILLO (DIstiLLed Language-ActiOn World Model), a fast steering layer that shifts the paradigm from "simulate-then-act" to "describe-then-act." DILLO is trained via cross-modal distillation, where a privileged Vision Language Model teacher annotates offline trajectories and a latent-conditioned Large Language Model student learns to predict semantic outcomes. This creates a text-only inference path, bypassing heavy visual generation entirely, achieving a 14x speedup over baselines. Experiments on MetaWorld and LIBERO demonstrate that DILLO produces high-fidelity descriptions of the next state and is able to steer the policy, improving episode success rate by up to 15 pp and 9.3 pp on average across tasks. |
| title | Describe-Then-Act: Proactive Agent Steering via Distilled Language-Action World Models |
| topic | Artificial Intelligence |
| url | https://arxiv.org/abs/2603.23149 |