SeaPO: Strategic Error Amplification for Robust Preference Optimization of Large Language Models
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2025
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866914064132734976 |
|---|---|
| author | Rao, Jun Liao, Yunjie Liu, Xuebo Lin, Zepeng Lian, Lian Jin, Dong Cheng, Shengjun Yu, Jun Zhang, Min |
| author_facet | Rao, Jun Liao, Yunjie Liu, Xuebo Lin, Zepeng Lian, Lian Jin, Dong Cheng, Shengjun Yu, Jun Zhang, Min |
| contents | Existing alignment methods for preference optimization of large language models (LLMs) aim to enhance model performance by utilizing pairs of positive and negative samples. However, due to the limited capacity of models in scoring or generating responses, the quality of positive and negative samples may become similar during training, which complicates optimization for preference learning. To address this issue, we introduce SeaPO, a Strategic Error Amplification method that leverages three error types commonly occurring in LLMs to introduce specific error patterns into the model Preference Optimization. This strategy ensures that negative samples are more erroneous than positive samples and preference-based training is employed to mitigate the occurrence of these errors, thereby enhancing model performance. Evaluations across five capability dimensions and different model scales (1.5B to 14B) demonstrate that the generated data significantly improved overall model performance, particularly in terms of truthfulness, with improvements of 5-10 percentage points observed. Further analysis reveals that task performance varies depending on the error types introduced. Injecting the most common error types improves performance in related tasks, while a mix of error types leads to a broader performance enhancement: most tasks show stable improvements, while a few tasks exhibit significant gains. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2509_24781 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | SeaPO: Strategic Error Amplification for Robust Preference Optimization of Large Language Models Rao, Jun Liao, Yunjie Liu, Xuebo Lin, Zepeng Lian, Lian Jin, Dong Cheng, Shengjun Yu, Jun Zhang, Min Computation and Language Existing alignment methods for preference optimization of large language models (LLMs) aim to enhance model performance by utilizing pairs of positive and negative samples. However, due to the limited capacity of models in scoring or generating responses, the quality of positive and negative samples may become similar during training, which complicates optimization for preference learning. To address this issue, we introduce SeaPO, a Strategic Error Amplification method that leverages three error types commonly occurring in LLMs to introduce specific error patterns into the model Preference Optimization. This strategy ensures that negative samples are more erroneous than positive samples and preference-based training is employed to mitigate the occurrence of these errors, thereby enhancing model performance. Evaluations across five capability dimensions and different model scales (1.5B to 14B) demonstrate that the generated data significantly improved overall model performance, particularly in terms of truthfulness, with improvements of 5-10 percentage points observed. Further analysis reveals that task performance varies depending on the error types introduced. Injecting the most common error types improves performance in related tasks, while a mix of error types leads to a broader performance enhancement: most tasks show stable improvements, while a few tasks exhibit significant gains. |
| title | SeaPO: Strategic Error Amplification for Robust Preference Optimization of Large Language Models |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2509.24781 |