Gespeichert in:
| Hauptverfasser: | , , , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2026
|
| Schlagworte: | |
| Online-Zugang: | https://arxiv.org/abs/2605.14152 |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866910218735058944 |
|---|---|
| author | Lee, Michael S. Maurya, Yash Rein, Drew Herring, Bert Nguyen, Jonathan Song, Kyungho Sehwag, Udari Madhushani Cho, Jiyeon Deshpande, Kaustubh Jang, Yeongkyun Joo, Jiyeon Choi, Minn Seok Fuelle, Evi Knight, Christina Q Brandifino, Joseph Fenkell, Max |
| author_facet | Lee, Michael S. Maurya, Yash Rein, Drew Herring, Bert Nguyen, Jonathan Song, Kyungho Sehwag, Udari Madhushani Cho, Jiyeon Deshpande, Kaustubh Jang, Yeongkyun Joo, Jiyeon Choi, Minn Seok Fuelle, Evi Knight, Christina Q Brandifino, Joseph Fenkell, Max |
| contents | Safety evaluations for large language models (LLMs) increasingly target high-stakes National Security and Public Safety (NSPS) risks, yet multilingual safety is typically assessed through translation-only benchmarks that preserve the underlying scenario, and empirical evidence of how language and geopolitical context interact remains limited to a narrow set of language pairs. We introduce \emph{ROK-FORTRESS} https://huggingface.co/datasets/ScaleAI/ROK-FORTRESS_public, a bilingual, culturally adversarial NSPS benchmark that uses the English--Korean language pair and U.S.--ROK geopolitical axis as a case study, separating the effects of language and geopolitical grounding via a \emph{transcreation matrix}: adversarial intents are evaluated under controlled combinations of (i) English versus Korean language and (ii) U.S.\ versus Korean entities, institutions, and operational details. Each adversarial prompt is paired with a dual-use benign counterpart to quantify over-refusal. Model responses are then scored using calibrated LLM-as-a-judge panels, applying our expert-crafted, prompt-specific binary rubrics.
Across a dual-track set of frontier and Korean-optimized models, we find a consistent suppression effect in Korean variants and substantial model-to-model variation in how geopolitical grounding interacts with language. In many models, Korean grounding mitigates the Korean language-driven suppression -- with no model showing significant amplification in the other direction -- indicating that, at least in the English--Korean case, safety behavior is shaped by language-as-risk signals and context interactions that translation-only evaluations miss. The transcreation matrix methodology is designed to generalize to other language--culture pairs. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2605_14152 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | ROK-FORTRESS: Measuring the Effect of Geopolitical Transcreation for National Security and Public Safety Lee, Michael S. Maurya, Yash Rein, Drew Herring, Bert Nguyen, Jonathan Song, Kyungho Sehwag, Udari Madhushani Cho, Jiyeon Deshpande, Kaustubh Jang, Yeongkyun Joo, Jiyeon Choi, Minn Seok Fuelle, Evi Knight, Christina Q Brandifino, Joseph Fenkell, Max Computation and Language Artificial Intelligence Cryptography and Security Computers and Society Safety evaluations for large language models (LLMs) increasingly target high-stakes National Security and Public Safety (NSPS) risks, yet multilingual safety is typically assessed through translation-only benchmarks that preserve the underlying scenario, and empirical evidence of how language and geopolitical context interact remains limited to a narrow set of language pairs. We introduce \emph{ROK-FORTRESS} https://huggingface.co/datasets/ScaleAI/ROK-FORTRESS_public, a bilingual, culturally adversarial NSPS benchmark that uses the English--Korean language pair and U.S.--ROK geopolitical axis as a case study, separating the effects of language and geopolitical grounding via a \emph{transcreation matrix}: adversarial intents are evaluated under controlled combinations of (i) English versus Korean language and (ii) U.S.\ versus Korean entities, institutions, and operational details. Each adversarial prompt is paired with a dual-use benign counterpart to quantify over-refusal. Model responses are then scored using calibrated LLM-as-a-judge panels, applying our expert-crafted, prompt-specific binary rubrics. Across a dual-track set of frontier and Korean-optimized models, we find a consistent suppression effect in Korean variants and substantial model-to-model variation in how geopolitical grounding interacts with language. In many models, Korean grounding mitigates the Korean language-driven suppression -- with no model showing significant amplification in the other direction -- indicating that, at least in the English--Korean case, safety behavior is shaped by language-as-risk signals and context interactions that translation-only evaluations miss. The transcreation matrix methodology is designed to generalize to other language--culture pairs. |
| title | ROK-FORTRESS: Measuring the Effect of Geopolitical Transcreation for National Security and Public Safety |
| topic | Computation and Language Artificial Intelligence Cryptography and Security Computers and Society |
| url | https://arxiv.org/abs/2605.14152 |