LLMs Lost in Translation: M-ALERT uncovers Cross-Linguistic Safety Inconsistencies
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2024
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866918067117752320 |
|---|---|
| author | Friedrich, Felix Tedeschi, Simone Schramowski, Patrick Brack, Manuel Navigli, Roberto Nguyen, Huu Li, Bo Kersting, Kristian |
| author_facet | Friedrich, Felix Tedeschi, Simone Schramowski, Patrick Brack, Manuel Navigli, Roberto Nguyen, Huu Li, Bo Kersting, Kristian |
| contents | Building safe Large Language Models (LLMs) across multiple languages is essential in ensuring both safe access and linguistic diversity. To this end, we conduct a large-scale, comprehensive safety evaluation of the current LLM landscape. For this purpose, we introduce M-ALERT, a multilingual benchmark that evaluates the safety of LLMs in five languages: English, French, German, Italian, and Spanish. M-ALERT includes 15k high-quality prompts per language, totaling 75k, with category-wise annotations. Our extensive experiments on 39 state-of-the-art LLMs highlight the importance of language-specific safety analysis, revealing that models often exhibit significant inconsistencies in safety across languages and categories. For instance, Llama3.2 shows high unsafety in category crime_tax for Italian but remains safe in other languages. Similar inconsistencies can be observed across all models. In contrast, certain categories, such as substance_cannabis and crime_propaganda, consistently trigger unsafe responses across models and languages. These findings underscore the need for robust multilingual safety practices in LLMs to ensure responsible usage across diverse communities. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2412_15035 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | LLMs Lost in Translation: M-ALERT uncovers Cross-Linguistic Safety Inconsistencies Friedrich, Felix Tedeschi, Simone Schramowski, Patrick Brack, Manuel Navigli, Roberto Nguyen, Huu Li, Bo Kersting, Kristian Computation and Language Building safe Large Language Models (LLMs) across multiple languages is essential in ensuring both safe access and linguistic diversity. To this end, we conduct a large-scale, comprehensive safety evaluation of the current LLM landscape. For this purpose, we introduce M-ALERT, a multilingual benchmark that evaluates the safety of LLMs in five languages: English, French, German, Italian, and Spanish. M-ALERT includes 15k high-quality prompts per language, totaling 75k, with category-wise annotations. Our extensive experiments on 39 state-of-the-art LLMs highlight the importance of language-specific safety analysis, revealing that models often exhibit significant inconsistencies in safety across languages and categories. For instance, Llama3.2 shows high unsafety in category crime_tax for Italian but remains safe in other languages. Similar inconsistencies can be observed across all models. In contrast, certain categories, such as substance_cannabis and crime_propaganda, consistently trigger unsafe responses across models and languages. These findings underscore the need for robust multilingual safety practices in LLMs to ensure responsible usage across diverse communities. |
| title | LLMs Lost in Translation: M-ALERT uncovers Cross-Linguistic Safety Inconsistencies |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2412.15035 |