Saved in:
| Main Authors: | , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2506.22056 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866911026092441600 |
|---|---|
| author | Zhang, Xuan Jiang, Ziyan Meng, Rui Leng, Yifei Xiao, Zhenbang Wang, Zora Zhiruo Shang, Yanyi Kong, Dehan |
| author_facet | Zhang, Xuan Jiang, Ziyan Meng, Rui Leng, Yifei Xiao, Zhenbang Wang, Zora Zhiruo Shang, Yanyi Kong, Dehan |
| contents | Trajectory data, capturing human actions and environmental states across various modalities, holds significant potential for enhancing AI agent capabilities, particularly in GUI environments. However, how to model the representation of trajectory-level data presents a significant challenge that has not been systematically addressed amid explosive trajectory data growth. In this work, we introduce Multimodal Trajectory Retrieval, bridging the gap between universal retrieval and agent-centric trajectory modeling. We construct the Unified Agent Trajectory Dataset (UATD) from annotated demonstrations and states across diverse real-world scenarios. Based on this, we present GAE-Bench, a benchmark containing a large number of trajectory-based retrieval pairs. In addition, we propose GAE-Retriever, a multimodal retrieval framework that adopts vision-language models and incorporates optimized contrastive learning through a token selection and the GradCache mechanism. Comprehensive evaluations across multiple datasets show that GAE-Retriever consistently outperforms strong baselines in retrieval recall, highlighting its effectiveness in advancing multimodal trajectory retrieval. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2506_22056 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Universal Retrieval for Multimodal Trajectory Modeling Zhang, Xuan Jiang, Ziyan Meng, Rui Leng, Yifei Xiao, Zhenbang Wang, Zora Zhiruo Shang, Yanyi Kong, Dehan Artificial Intelligence Trajectory data, capturing human actions and environmental states across various modalities, holds significant potential for enhancing AI agent capabilities, particularly in GUI environments. However, how to model the representation of trajectory-level data presents a significant challenge that has not been systematically addressed amid explosive trajectory data growth. In this work, we introduce Multimodal Trajectory Retrieval, bridging the gap between universal retrieval and agent-centric trajectory modeling. We construct the Unified Agent Trajectory Dataset (UATD) from annotated demonstrations and states across diverse real-world scenarios. Based on this, we present GAE-Bench, a benchmark containing a large number of trajectory-based retrieval pairs. In addition, we propose GAE-Retriever, a multimodal retrieval framework that adopts vision-language models and incorporates optimized contrastive learning through a token selection and the GradCache mechanism. Comprehensive evaluations across multiple datasets show that GAE-Retriever consistently outperforms strong baselines in retrieval recall, highlighting its effectiveness in advancing multimodal trajectory retrieval. |
| title | Universal Retrieval for Multimodal Trajectory Modeling |
| topic | Artificial Intelligence |
| url | https://arxiv.org/abs/2506.22056 |