Evaluating Implicit Regulatory Compliance in LLM Tool Invocation via Logic-Guided Synthesis
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2026
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866914251062378496 |
|---|---|
| author | Song, Da Huang, Yuheng Chen, Boqi Cong, Tianshuo Goebel, Randy Ma, Lei Khomh, Foutse |
| author_facet | Song, Da Huang, Yuheng Chen, Boqi Cong, Tianshuo Goebel, Randy Ma, Lei Khomh, Foutse |
| contents | The integration of large language models (LLMs) into autonomous agents has enabled complex tool use, yet in high-stakes domains, these systems must strictly adhere to regulatory standards beyond simple functional correctness. However, existing benchmarks often overlook implicit regulatory compliance, thus failing to evaluate whether LLMs can autonomously enforce mandatory safety constraints. To fill this gap, we introduce LogiSafetyGen, a framework that converts unstructured regulations into Linear Temporal Logic oracles and employs logic-guided fuzzing to synthesize valid, safety-critical traces. Building on this framework, we construct LogiSafetyBench, a benchmark comprising 240 human-verified tasks that require LLMs to generate Python programs that satisfy both functional objectives and latent compliance rules. Evaluations of 13 state-of-the-art (SOTA) LLMs reveal that larger models, despite achieving better functional correctness, frequently prioritize task completion over safety, which results in non-compliant behavior. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2601_08196 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | Evaluating Implicit Regulatory Compliance in LLM Tool Invocation via Logic-Guided Synthesis Song, Da Huang, Yuheng Chen, Boqi Cong, Tianshuo Goebel, Randy Ma, Lei Khomh, Foutse Computation and Language Artificial Intelligence Cryptography and Security Logic in Computer Science Software Engineering The integration of large language models (LLMs) into autonomous agents has enabled complex tool use, yet in high-stakes domains, these systems must strictly adhere to regulatory standards beyond simple functional correctness. However, existing benchmarks often overlook implicit regulatory compliance, thus failing to evaluate whether LLMs can autonomously enforce mandatory safety constraints. To fill this gap, we introduce LogiSafetyGen, a framework that converts unstructured regulations into Linear Temporal Logic oracles and employs logic-guided fuzzing to synthesize valid, safety-critical traces. Building on this framework, we construct LogiSafetyBench, a benchmark comprising 240 human-verified tasks that require LLMs to generate Python programs that satisfy both functional objectives and latent compliance rules. Evaluations of 13 state-of-the-art (SOTA) LLMs reveal that larger models, despite achieving better functional correctness, frequently prioritize task completion over safety, which results in non-compliant behavior. |
| title | Evaluating Implicit Regulatory Compliance in LLM Tool Invocation via Logic-Guided Synthesis |
| topic | Computation and Language Artificial Intelligence Cryptography and Security Logic in Computer Science Software Engineering |
| url | https://arxiv.org/abs/2601.08196 |