Astra: Efficient Transformer Architecture and Contrastive Dynamics Learning for Embodied Instruction Following
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2024
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866912832344293376 |
|---|---|
| author | Ma, Yueen Chi, Dafeng Wu, Shiguang Liu, Yuecheng Zhuang, Yuzheng King, Irwin |
| author_facet | Ma, Yueen Chi, Dafeng Wu, Shiguang Liu, Yuecheng Zhuang, Yuzheng King, Irwin |
| contents | Vision-language-action models have gained significant attention for their ability to model multimodal sequences in embodied instruction following tasks. However, most existing models rely on causal attention, which we find suboptimal for processing sequences composed of interleaved segments from different modalities. In this paper, we introduce Astra, a novel Transformer architecture featuring trajectory attention and learnable action queries, designed to efficiently process segmented multimodal trajectories and predict actions for imitation learning. Furthermore, we propose a contrastive dynamics learning objective to enhance the model's understanding of environment dynamics and multimodal alignment, complementing the primary behavior cloning objective. Through extensive experiments on three large-scale robot manipulation benchmarks, Astra demonstrates substantial performance improvements over previous models. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2408_01147 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Astra: Efficient Transformer Architecture and Contrastive Dynamics Learning for Embodied Instruction Following Ma, Yueen Chi, Dafeng Wu, Shiguang Liu, Yuecheng Zhuang, Yuzheng King, Irwin Robotics Vision-language-action models have gained significant attention for their ability to model multimodal sequences in embodied instruction following tasks. However, most existing models rely on causal attention, which we find suboptimal for processing sequences composed of interleaved segments from different modalities. In this paper, we introduce Astra, a novel Transformer architecture featuring trajectory attention and learnable action queries, designed to efficiently process segmented multimodal trajectories and predict actions for imitation learning. Furthermore, we propose a contrastive dynamics learning objective to enhance the model's understanding of environment dynamics and multimodal alignment, complementing the primary behavior cloning objective. Through extensive experiments on three large-scale robot manipulation benchmarks, Astra demonstrates substantial performance improvements over previous models. |
| title | Astra: Efficient Transformer Architecture and Contrastive Dynamics Learning for Embodied Instruction Following |
| topic | Robotics |
| url | https://arxiv.org/abs/2408.01147 |