MPO: Multilingual Safety Alignment via Reward Gap Optimization
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2025
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866915298807906304 |
|---|---|
| author | Zhao, Weixiang Hu, Yulin Deng, Yang Wu, Tongtong Zhang, Wenxuan Guo, Jiahe Zhang, An Zhao, Yanyan Qin, Bing Chua, Tat-Seng Liu, Ting |
| author_facet | Zhao, Weixiang Hu, Yulin Deng, Yang Wu, Tongtong Zhang, Wenxuan Guo, Jiahe Zhang, An Zhao, Yanyan Qin, Bing Chua, Tat-Seng Liu, Ting |
| contents | Large language models (LLMs) have become increasingly central to AI applications worldwide, necessitating robust multilingual safety alignment to ensure secure deployment across diverse linguistic contexts. Existing preference learning methods for safety alignment, such as RLHF and DPO, are primarily monolingual and struggle with noisy multilingual data. To address these limitations, we introduce Multilingual reward gaP Optimization (MPO), a novel approach that leverages the well-aligned safety capabilities of the dominant language (English) to improve safety alignment across multiple languages. MPO directly minimizes the reward gap difference between the dominant language and target languages, effectively transferring safety capabilities while preserving the original strengths of the dominant language. Extensive experiments on three LLMs, LLaMA-3.1, Gemma-2 and Qwen2.5, validate MPO's efficacy in multilingual safety alignment without degrading general multilingual utility. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2505_16869 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | MPO: Multilingual Safety Alignment via Reward Gap Optimization Zhao, Weixiang Hu, Yulin Deng, Yang Wu, Tongtong Zhang, Wenxuan Guo, Jiahe Zhang, An Zhao, Yanyan Qin, Bing Chua, Tat-Seng Liu, Ting Computation and Language Large language models (LLMs) have become increasingly central to AI applications worldwide, necessitating robust multilingual safety alignment to ensure secure deployment across diverse linguistic contexts. Existing preference learning methods for safety alignment, such as RLHF and DPO, are primarily monolingual and struggle with noisy multilingual data. To address these limitations, we introduce Multilingual reward gaP Optimization (MPO), a novel approach that leverages the well-aligned safety capabilities of the dominant language (English) to improve safety alignment across multiple languages. MPO directly minimizes the reward gap difference between the dominant language and target languages, effectively transferring safety capabilities while preserving the original strengths of the dominant language. Extensive experiments on three LLMs, LLaMA-3.1, Gemma-2 and Qwen2.5, validate MPO's efficacy in multilingual safety alignment without degrading general multilingual utility. |
| title | MPO: Multilingual Safety Alignment via Reward Gap Optimization |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2505.16869 |