Reinforcement Learning with Semantic Rewards Enables Low-Resource Language Expansion without Alignment Tax
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2026
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866918501123358720 |
|---|---|
| author | Su, Zeli Zhang, Ziyin Liu, Zhou Song, Xuexian Xu, Zhankai Zheng, Longfei Zhang, Xiaolu Fu, Rong Xu, Guixian Zhang, Wentao |
| author_facet | Su, Zeli Zhang, Ziyin Liu, Zhou Song, Xuexian Xu, Zhankai Zheng, Longfei Zhang, Xiaolu Fu, Rong Xu, Guixian Zhang, Wentao |
| contents | Extending large language models (LLMs) to low-resource languages often incurs an "alignment tax": improvements in the target language come at the cost of catastrophic forgetting in general capabilities. We argue that this trade-off arises from the rigidity of supervised fine-tuning (SFT), which enforces token-level surface imitation on narrow and biased data distributions. To address this limitation, we propose a semantic-space alignment paradigm powered by Group Relative Policy Optimization (GRPO), where the model is optimized using embedding-level semantic rewards rather than likelihood maximization. This objective encourages meaning preservation through flexible realizations, enabling controlled updates that reduce destructive interference with pretrained knowledge. We evaluate our approach on Tibetan-Chinese machine translation and Tibetan headline generation. Experiments show that our method acquires low-resource capabilities while markedly mitigating alignment tax, preserving general competence more effectively than SFT. Despite producing less rigid surface overlap, semantic RL yields higher semantic quality and preference in open-ended generation, and few-shot transfer results indicate that it learns more transferable and robust representations under limited supervision. Overall, our study demonstrates that reinforcement learning with semantic rewards provides a safer and more reliable pathway for inclusive low-resource language expansion. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2605_14366 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | Reinforcement Learning with Semantic Rewards Enables Low-Resource Language Expansion without Alignment Tax Su, Zeli Zhang, Ziyin Liu, Zhou Song, Xuexian Xu, Zhankai Zheng, Longfei Zhang, Xiaolu Fu, Rong Xu, Guixian Zhang, Wentao Computation and Language Machine Learning Extending large language models (LLMs) to low-resource languages often incurs an "alignment tax": improvements in the target language come at the cost of catastrophic forgetting in general capabilities. We argue that this trade-off arises from the rigidity of supervised fine-tuning (SFT), which enforces token-level surface imitation on narrow and biased data distributions. To address this limitation, we propose a semantic-space alignment paradigm powered by Group Relative Policy Optimization (GRPO), where the model is optimized using embedding-level semantic rewards rather than likelihood maximization. This objective encourages meaning preservation through flexible realizations, enabling controlled updates that reduce destructive interference with pretrained knowledge. We evaluate our approach on Tibetan-Chinese machine translation and Tibetan headline generation. Experiments show that our method acquires low-resource capabilities while markedly mitigating alignment tax, preserving general competence more effectively than SFT. Despite producing less rigid surface overlap, semantic RL yields higher semantic quality and preference in open-ended generation, and few-shot transfer results indicate that it learns more transferable and robust representations under limited supervision. Overall, our study demonstrates that reinforcement learning with semantic rewards provides a safer and more reliable pathway for inclusive low-resource language expansion. |
| title | Reinforcement Learning with Semantic Rewards Enables Low-Resource Language Expansion without Alignment Tax |
| topic | Computation and Language Machine Learning |
| url | https://arxiv.org/abs/2605.14366 |