TL-Training: A Task-Feature-Based Framework for Training Large Language Models in Tool Use
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866916918742482944 |
|---|---|
| author | Ye, Junjie Wu, Yilong Li, Sixian Yang, Yuming Xi, Zhiheng Gui, Tao Zhang, Qi Huang, Xuanjing Wang, Peng Shi, Zhongchao Fan, Jianping Du, Zhengyin |
| author_facet | Ye, Junjie Wu, Yilong Li, Sixian Yang, Yuming Xi, Zhiheng Gui, Tao Zhang, Qi Huang, Xuanjing Wang, Peng Shi, Zhongchao Fan, Jianping Du, Zhengyin |
| contents | Large language models (LLMs) achieve remarkable advancements by leveraging tools to interact with environments, a critical step toward generalized AI. However, the standard supervised fine-tuning (SFT) approach, which relies on large-scale datasets, often overlooks task-specific characteristics in tool use, leading to performance bottlenecks. To address this issue, we analyze three existing LLMs and uncover key insights: training data can inadvertently impede tool-use behavior, token importance is distributed unevenly, and errors in tool calls fall into a small set of categories. Building on these findings, we propose~\emph{TL-Training}, a task-feature-based framework that mitigates the effects of suboptimal training data, dynamically adjusts token weights to prioritize key tokens during SFT, and incorporates a robust reward mechanism tailored to error categories, optimized through proximal policy optimization. We validate TL-Training by training CodeLLaMA-2-7B and evaluating it on four open-source test sets. Our results demonstrate that the LLM trained by our method matches or surpasses both open- and closed-source LLMs in tool-use performance using only 1,217 training data points. Additionally, our method enhances robustness in noisy environments and improves general task performance, offering a scalable and efficient paradigm for tool-use training in LLMs. Code and data are available at https://github.com/Junjie-Ye/TL-Training. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2412_15495 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | TL-Training: A Task-Feature-Based Framework for Training Large Language Models in Tool Use Ye, Junjie Wu, Yilong Li, Sixian Yang, Yuming Xi, Zhiheng Gui, Tao Zhang, Qi Huang, Xuanjing Wang, Peng Shi, Zhongchao Fan, Jianping Du, Zhengyin Computation and Language Artificial Intelligence Large language models (LLMs) achieve remarkable advancements by leveraging tools to interact with environments, a critical step toward generalized AI. However, the standard supervised fine-tuning (SFT) approach, which relies on large-scale datasets, often overlooks task-specific characteristics in tool use, leading to performance bottlenecks. To address this issue, we analyze three existing LLMs and uncover key insights: training data can inadvertently impede tool-use behavior, token importance is distributed unevenly, and errors in tool calls fall into a small set of categories. Building on these findings, we propose~\emph{TL-Training}, a task-feature-based framework that mitigates the effects of suboptimal training data, dynamically adjusts token weights to prioritize key tokens during SFT, and incorporates a robust reward mechanism tailored to error categories, optimized through proximal policy optimization. We validate TL-Training by training CodeLLaMA-2-7B and evaluating it on four open-source test sets. Our results demonstrate that the LLM trained by our method matches or surpasses both open- and closed-source LLMs in tool-use performance using only 1,217 training data points. Additionally, our method enhances robustness in noisy environments and improves general task performance, offering a scalable and efficient paradigm for tool-use training in LLMs. Code and data are available at https://github.com/Junjie-Ye/TL-Training. |
| title | TL-Training: A Task-Feature-Based Framework for Training Large Language Models in Tool Use |
| topic | Computation and Language Artificial Intelligence |
| url | https://arxiv.org/abs/2412.15495 |