CONGRAD:Conflicting Gradient Filtering for Multilingual Preference Alignment
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2025
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866914148620697600 |
|---|---|
| author | Li, Jiangnan Vu, Thuy-Trang Herold, Christian Tebbifakhr, Amirhossein Khadivi, Shahram Haffari, Gholamreza |
| author_facet | Li, Jiangnan Vu, Thuy-Trang Herold, Christian Tebbifakhr, Amirhossein Khadivi, Shahram Haffari, Gholamreza |
| contents | Naive joint training of large language models (LLMs) for multilingual preference alignment can suffer from negative interference. This is a known issue in multilingual training, where conflicting objectives degrade overall performance. However, the impact of this phenomenon in the context of multilingual preference alignment remains largely underexplored. To address this issue, we propose CONGRAD, a scalable and effective filtering method that selects high-quality preference samples with minimal gradient conflicts across languages. Our method leverages gradient surgery to retain samples aligned with an aggregated multilingual update direction. Additionally, we incorporate a sublinear gradient compression strategy that reduces memory overhead during gradient accumulation. We integrate CONGRAD into self-rewarding framework and evaluate on LLaMA3-8B and Gemma2-2B across 10 languages. Results show that CONGRAD consistently outperforms strong baselines in both seen and unseen languages, with minimal alignment tax. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2503_23777 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | CONGRAD:Conflicting Gradient Filtering for Multilingual Preference Alignment Li, Jiangnan Vu, Thuy-Trang Herold, Christian Tebbifakhr, Amirhossein Khadivi, Shahram Haffari, Gholamreza Computation and Language Naive joint training of large language models (LLMs) for multilingual preference alignment can suffer from negative interference. This is a known issue in multilingual training, where conflicting objectives degrade overall performance. However, the impact of this phenomenon in the context of multilingual preference alignment remains largely underexplored. To address this issue, we propose CONGRAD, a scalable and effective filtering method that selects high-quality preference samples with minimal gradient conflicts across languages. Our method leverages gradient surgery to retain samples aligned with an aggregated multilingual update direction. Additionally, we incorporate a sublinear gradient compression strategy that reduces memory overhead during gradient accumulation. We integrate CONGRAD into self-rewarding framework and evaluate on LLaMA3-8B and Gemma2-2B across 10 languages. Results show that CONGRAD consistently outperforms strong baselines in both seen and unseen languages, with minimal alignment tax. |
| title | CONGRAD:Conflicting Gradient Filtering for Multilingual Preference Alignment |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2503.23777 |