Transferable Post-training via Inverse Value Learning
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866913890650030080 |
|---|---|
| author | Lu, Xinyu Wen, Xueru Lu, Yaojie Yu, Bowen Lin, Hongyu Yu, Haiyang Sun, Le Han, Xianpei Li, Yongbin |
| author_facet | Lu, Xinyu Wen, Xueru Lu, Yaojie Yu, Bowen Lin, Hongyu Yu, Haiyang Sun, Le Han, Xianpei Li, Yongbin |
| contents | As post-training processes utilize increasingly large datasets and base models continue to grow in size, the computational demands and implementation challenges of existing algorithms are escalating significantly. In this paper, we propose modeling the changes at the logits level during post-training using a separate neural network (i.e., the value network). After training this network on a small base model using demonstrations, this network can be seamlessly integrated with other pre-trained models during inference, enables them to achieve similar capability enhancements. We systematically investigate the best practices for this paradigm in terms of pre-training weights and connection schemes. We demonstrate that the resulting value network has broad transferability across pre-trained models of different parameter sizes within the same family, models undergoing continuous pre-training within the same family, and models with different vocabularies across families. In certain cases, it can achieve performance comparable to full-parameter fine-tuning. Furthermore, we explore methods to enhance the transferability of the value model and prevent overfitting to the base model used during training. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2410_21027 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Transferable Post-training via Inverse Value Learning Lu, Xinyu Wen, Xueru Lu, Yaojie Yu, Bowen Lin, Hongyu Yu, Haiyang Sun, Le Han, Xianpei Li, Yongbin Machine Learning Computation and Language As post-training processes utilize increasingly large datasets and base models continue to grow in size, the computational demands and implementation challenges of existing algorithms are escalating significantly. In this paper, we propose modeling the changes at the logits level during post-training using a separate neural network (i.e., the value network). After training this network on a small base model using demonstrations, this network can be seamlessly integrated with other pre-trained models during inference, enables them to achieve similar capability enhancements. We systematically investigate the best practices for this paradigm in terms of pre-training weights and connection schemes. We demonstrate that the resulting value network has broad transferability across pre-trained models of different parameter sizes within the same family, models undergoing continuous pre-training within the same family, and models with different vocabularies across families. In certain cases, it can achieve performance comparable to full-parameter fine-tuning. Furthermore, we explore methods to enhance the transferability of the value model and prevent overfitting to the base model used during training. |
| title | Transferable Post-training via Inverse Value Learning |
| topic | Machine Learning Computation and Language |
| url | https://arxiv.org/abs/2410.21027 |