COMPASS: A Framework for Evaluating Organization-Specific Policy Alignment in LLMs
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2026
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866912801895743488 |
|---|---|
| author | Choi, Dasol Lee, DongGeon Kartono, Brigitta Jesica Berndt, Helena Kwon, Taeyoun Jang, Joonwon Park, Haon Yu, Hwanjo Kahng, Minsuk |
| author_facet | Choi, Dasol Lee, DongGeon Kartono, Brigitta Jesica Berndt, Helena Kwon, Taeyoun Jang, Joonwon Park, Haon Yu, Hwanjo Kahng, Minsuk |
| contents | As large language models are deployed in high-stakes enterprise applications, from healthcare to finance, ensuring adherence to organization-specific policies has become essential. Yet existing safety evaluations focus exclusively on universal harms. We present COMPASS (Company/Organization Policy Alignment Assessment), the first systematic framework for evaluating whether LLMs comply with organizational allowlist and denylist policies. We apply COMPASS to eight diverse industry scenarios, generating and validating 5,920 queries that test both routine compliance and adversarial robustness through strategically designed edge cases. Evaluating seven state-of-the-art models, we uncover a fundamental asymmetry: models reliably handle legitimate requests (>95% accuracy) but catastrophically fail at enforcing prohibitions, refusing only 13-40% of adversarial denylist violations. These results demonstrate that current LLMs lack the robustness required for policy-critical deployments, establishing COMPASS as an essential evaluation framework for organizational AI safety. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2601_01836 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | COMPASS: A Framework for Evaluating Organization-Specific Policy Alignment in LLMs Choi, Dasol Lee, DongGeon Kartono, Brigitta Jesica Berndt, Helena Kwon, Taeyoun Jang, Joonwon Park, Haon Yu, Hwanjo Kahng, Minsuk Artificial Intelligence Computers and Society As large language models are deployed in high-stakes enterprise applications, from healthcare to finance, ensuring adherence to organization-specific policies has become essential. Yet existing safety evaluations focus exclusively on universal harms. We present COMPASS (Company/Organization Policy Alignment Assessment), the first systematic framework for evaluating whether LLMs comply with organizational allowlist and denylist policies. We apply COMPASS to eight diverse industry scenarios, generating and validating 5,920 queries that test both routine compliance and adversarial robustness through strategically designed edge cases. Evaluating seven state-of-the-art models, we uncover a fundamental asymmetry: models reliably handle legitimate requests (>95% accuracy) but catastrophically fail at enforcing prohibitions, refusing only 13-40% of adversarial denylist violations. These results demonstrate that current LLMs lack the robustness required for policy-critical deployments, establishing COMPASS as an essential evaluation framework for organizational AI safety. |
| title | COMPASS: A Framework for Evaluating Organization-Specific Policy Alignment in LLMs |
| topic | Artificial Intelligence Computers and Society |
| url | https://arxiv.org/abs/2601.01836 |