Disentangling Reasoning Tokens and Boilerplate Tokens For Language Model Fine-tuning
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866916532736491520 |
|---|---|
| author | Ye, Ziang Zhang, Zhenru Zhang, Yang Ma, Jianxin Lin, Junyang Feng, Fuli |
| author_facet | Ye, Ziang Zhang, Zhenru Zhang, Yang Ma, Jianxin Lin, Junyang Feng, Fuli |
| contents | When using agent-task datasets to enhance agent capabilities for Large Language Models (LLMs), current methodologies often treat all tokens within a sample equally. However, we argue that tokens serving different roles - specifically, reasoning tokens versus boilerplate tokens (e.g., those governing output format) - differ significantly in importance and learning complexity, necessitating their disentanglement and distinct treatment. To address this, we propose a novel Shuffle-Aware Discriminator (SHAD) for adaptive token discrimination. SHAD classifies tokens by exploiting predictability differences observed after shuffling input-output combinations across samples: boilerplate tokens, due to their repetitive nature among samples, maintain predictability, whereas reasoning tokens do not. Using SHAD, we propose the Reasoning-highlighted Fine-Tuning (RFT) method, which adaptively emphasizes reasoning tokens during fine-tuning, yielding notable performance gains over common Supervised Fine-Tuning (SFT). |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2412_14780 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Disentangling Reasoning Tokens and Boilerplate Tokens For Language Model Fine-tuning Ye, Ziang Zhang, Zhenru Zhang, Yang Ma, Jianxin Lin, Junyang Feng, Fuli Computation and Language When using agent-task datasets to enhance agent capabilities for Large Language Models (LLMs), current methodologies often treat all tokens within a sample equally. However, we argue that tokens serving different roles - specifically, reasoning tokens versus boilerplate tokens (e.g., those governing output format) - differ significantly in importance and learning complexity, necessitating their disentanglement and distinct treatment. To address this, we propose a novel Shuffle-Aware Discriminator (SHAD) for adaptive token discrimination. SHAD classifies tokens by exploiting predictability differences observed after shuffling input-output combinations across samples: boilerplate tokens, due to their repetitive nature among samples, maintain predictability, whereas reasoning tokens do not. Using SHAD, we propose the Reasoning-highlighted Fine-Tuning (RFT) method, which adaptively emphasizes reasoning tokens during fine-tuning, yielding notable performance gains over common Supervised Fine-Tuning (SFT). |
| title | Disentangling Reasoning Tokens and Boilerplate Tokens For Language Model Fine-tuning |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2412.14780 |