Saved in:
| Main Authors: | , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2410.17599 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866916449451245568 |
|---|---|
| author | Wu, Jiayi Sun, Hao Cai, Hengyi Su, Lixin Wang, Shuaiqiang Yin, Dawei Li, Xiang Gao, Ming |
| author_facet | Wu, Jiayi Sun, Hao Cai, Hengyi Su, Lixin Wang, Shuaiqiang Yin, Dawei Li, Xiang Gao, Ming |
| contents | The number of large language models (LLMs) with varying parameter scales and vocabularies is increasing. While they deliver powerful performance, they also face a set of common optimization needs to meet specific requirements or standards, such as instruction following or avoiding the output of sensitive information from the real world. However, how to reuse the fine-tuning outcomes of one model to other models to reduce training costs remains a challenge. To bridge this gap, we introduce Cross-model Control (CMC), a method that improves multiple LLMs in one-time training with a portable tiny language model. Specifically, we have observed that the logit shift before and after fine-tuning is remarkably similar across different models. Based on this insight, we incorporate a tiny language model with a minimal number of parameters. By training alongside a frozen template LLM, the tiny model gains the capability to alter the logits output by the LLMs. To make this tiny language model applicable to models with different vocabularies, we propose a novel token mapping strategy named PM-MinED. We have conducted extensive experiments on instruction tuning and unlearning tasks, demonstrating the effectiveness of CMC. Our code is available at https://github.com/wujwyi/CMC. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2410_17599 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Cross-model Control: Improving Multiple Large Language Models in One-time Training Wu, Jiayi Sun, Hao Cai, Hengyi Su, Lixin Wang, Shuaiqiang Yin, Dawei Li, Xiang Gao, Ming Computation and Language The number of large language models (LLMs) with varying parameter scales and vocabularies is increasing. While they deliver powerful performance, they also face a set of common optimization needs to meet specific requirements or standards, such as instruction following or avoiding the output of sensitive information from the real world. However, how to reuse the fine-tuning outcomes of one model to other models to reduce training costs remains a challenge. To bridge this gap, we introduce Cross-model Control (CMC), a method that improves multiple LLMs in one-time training with a portable tiny language model. Specifically, we have observed that the logit shift before and after fine-tuning is remarkably similar across different models. Based on this insight, we incorporate a tiny language model with a minimal number of parameters. By training alongside a frozen template LLM, the tiny model gains the capability to alter the logits output by the LLMs. To make this tiny language model applicable to models with different vocabularies, we propose a novel token mapping strategy named PM-MinED. We have conducted extensive experiments on instruction tuning and unlearning tasks, demonstrating the effectiveness of CMC. Our code is available at https://github.com/wujwyi/CMC. |
| title | Cross-model Control: Improving Multiple Large Language Models in One-time Training |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2410.17599 |