Saved in:
| Main Authors: | , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2502.05400 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866918388393050112 |
|---|---|
| author | Yang, Haoyan Le, Khiem Hua, Ting Gao, Shangqian Xu, Binfeng Tang, Zheng Xu, Jie Chawla, Nitesh V. Jin, Hongxia Srinivasan, Vijay |
| author_facet | Yang, Haoyan Le, Khiem Hua, Ting Gao, Shangqian Xu, Binfeng Tang, Zheng Xu, Jie Chawla, Nitesh V. Jin, Hongxia Srinivasan, Vijay |
| contents | Although LLMs have achieved significant success, their reliance on large volumes of human-annotated data has limited their potential for further scaling. In this situation, utilizing self-generated synthetic data has become crucial for fine-tuning LLMs without extensive human annotation. However, current methods often fail to ensure consistent improvements across iterations, with performance stagnating after only minimal updates. To overcome these challenges, we introduce Dynamic Noise Preference Optimization (DNPO), which combines dynamic sample labeling for constructing preference pairs with controlled, trainable noise injection during preference optimization. Our approach effectively prevents stagnation and enables continuous improvement. In experiments with Llama-3.2-3B and Zephyr-7B, DNPO consistently outperforms existing methods across multiple benchmarks. Additionally, with Zephyr-7B, DNPO shows a significant improvement in model-generated data quality, with a 29.4% win-loss rate gap compared to the baseline in GPT-4 evaluations. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2502_05400 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Dynamic Noise Preference Optimization: Self-Improvement of Large Language Models with Self-Synthetic Data Yang, Haoyan Le, Khiem Hua, Ting Gao, Shangqian Xu, Binfeng Tang, Zheng Xu, Jie Chawla, Nitesh V. Jin, Hongxia Srinivasan, Vijay Computation and Language Although LLMs have achieved significant success, their reliance on large volumes of human-annotated data has limited their potential for further scaling. In this situation, utilizing self-generated synthetic data has become crucial for fine-tuning LLMs without extensive human annotation. However, current methods often fail to ensure consistent improvements across iterations, with performance stagnating after only minimal updates. To overcome these challenges, we introduce Dynamic Noise Preference Optimization (DNPO), which combines dynamic sample labeling for constructing preference pairs with controlled, trainable noise injection during preference optimization. Our approach effectively prevents stagnation and enables continuous improvement. In experiments with Llama-3.2-3B and Zephyr-7B, DNPO consistently outperforms existing methods across multiple benchmarks. Additionally, with Zephyr-7B, DNPO shows a significant improvement in model-generated data quality, with a 29.4% win-loss rate gap compared to the baseline in GPT-4 evaluations. |
| title | Dynamic Noise Preference Optimization: Self-Improvement of Large Language Models with Self-Synthetic Data |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2502.05400 |