Multi-modal Agent Tuning: Building a VLM-Driven Agent for Efficient Tool Usage
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866915134090248192 |
|---|---|
| author | Gao, Zhi Zhang, Bofei Li, Pengxiang Ma, Xiaojian Yuan, Tao Fan, Yue Wu, Yuwei Jia, Yunde Zhu, Song-Chun Li, Qing |
| author_facet | Gao, Zhi Zhang, Bofei Li, Pengxiang Ma, Xiaojian Yuan, Tao Fan, Yue Wu, Yuwei Jia, Yunde Zhu, Song-Chun Li, Qing |
| contents | The advancement of large language models (LLMs) prompts the development of multi-modal agents, which are used as a controller to call external tools, providing a feasible way to solve practical tasks. In this paper, we propose a multi-modal agent tuning method that automatically generates multi-modal tool-usage data and tunes a vision-language model (VLM) as the controller for powerful tool-usage reasoning. To preserve the data quality, we prompt the GPT-4o mini model to generate queries, files, and trajectories, followed by query-file and trajectory verifiers. Based on the data synthesis pipeline, we collect the MM-Traj dataset that contains 20K tasks with trajectories of tool usage. Then, we develop the T3-Agent via \underline{T}rajectory \underline{T}uning on VLMs for \underline{T}ool usage using MM-Traj. Evaluations on the GTA and GAIA benchmarks show that the T3-Agent consistently achieves improvements on two popular VLMs: MiniCPM-V-8.5B and {Qwen2-VL-7B}, which outperforms untrained VLMs by $20\%$, showing the effectiveness of the proposed data synthesis pipeline, leading to high-quality data for tool-usage capabilities. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2412_15606 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Multi-modal Agent Tuning: Building a VLM-Driven Agent for Efficient Tool Usage Gao, Zhi Zhang, Bofei Li, Pengxiang Ma, Xiaojian Yuan, Tao Fan, Yue Wu, Yuwei Jia, Yunde Zhu, Song-Chun Li, Qing Artificial Intelligence Computer Vision and Pattern Recognition The advancement of large language models (LLMs) prompts the development of multi-modal agents, which are used as a controller to call external tools, providing a feasible way to solve practical tasks. In this paper, we propose a multi-modal agent tuning method that automatically generates multi-modal tool-usage data and tunes a vision-language model (VLM) as the controller for powerful tool-usage reasoning. To preserve the data quality, we prompt the GPT-4o mini model to generate queries, files, and trajectories, followed by query-file and trajectory verifiers. Based on the data synthesis pipeline, we collect the MM-Traj dataset that contains 20K tasks with trajectories of tool usage. Then, we develop the T3-Agent via \underline{T}rajectory \underline{T}uning on VLMs for \underline{T}ool usage using MM-Traj. Evaluations on the GTA and GAIA benchmarks show that the T3-Agent consistently achieves improvements on two popular VLMs: MiniCPM-V-8.5B and {Qwen2-VL-7B}, which outperforms untrained VLMs by $20\%$, showing the effectiveness of the proposed data synthesis pipeline, leading to high-quality data for tool-usage capabilities. |
| title | Multi-modal Agent Tuning: Building a VLM-Driven Agent for Efficient Tool Usage |
| topic | Artificial Intelligence Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2412.15606 |