PolicyLLM: Towards Excellent Comprehension of Public Policy for Large Language Models
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2026
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866918446405517312 |
|---|---|
| author | Bao, Han Zhang, Penghao Huang, Yue Yuan, Zhengqing Ru, Yanchi Su, Rui Zhou, Yujun Wang, Xiangqi Guo, Kehan Chawla, Nitesh V Ye, Yanfang Zhang, Xiangliang |
| author_facet | Bao, Han Zhang, Penghao Huang, Yue Yuan, Zhengqing Ru, Yanchi Su, Rui Zhou, Yujun Wang, Xiangqi Guo, Kehan Chawla, Nitesh V Ye, Yanfang Zhang, Xiangliang |
| contents | Large Language Models (LLMs) are increasingly integrated into real-world decision-making, including in the domain of public policy. Yet, their ability to comprehend and reason about policy-related content remains underexplored. To fill this gap, we present \textbf{\textit{PolicyBench}}, the first large-scale cross-system benchmark (US-China) evaluating policy comprehension, comprising 21K cases across a broad spectrum of policy areas, capturing the diversity and complexity of real-world governance. Following Bloom's taxonomy, the benchmark assesses three core capabilities: (1) \textbf{Memorization}: factual recall of policy knowledge, (2) \textbf{Understanding}: conceptual and contextual reasoning, and (3) \textbf{Application}: problem-solving in real-life policy scenarios. Building on this benchmark, we further propose \textbf{\textit{PolicyMoE}}, a domain-specialized Mixture-of-Experts (MoE) model with expert modules aligned to each cognitive level. The proposed models demonstrate stronger performance on application-oriented policy tasks than on memorization or conceptual understanding, and yields the highest accuracy on structured reasoning tasks. Our results reveal key limitations of current LLMs in policy understanding and suggest paths toward more reliable, policy-focused models. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2604_12995 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | PolicyLLM: Towards Excellent Comprehension of Public Policy for Large Language Models Bao, Han Zhang, Penghao Huang, Yue Yuan, Zhengqing Ru, Yanchi Su, Rui Zhou, Yujun Wang, Xiangqi Guo, Kehan Chawla, Nitesh V Ye, Yanfang Zhang, Xiangliang Computation and Language Computers and Society Large Language Models (LLMs) are increasingly integrated into real-world decision-making, including in the domain of public policy. Yet, their ability to comprehend and reason about policy-related content remains underexplored. To fill this gap, we present \textbf{\textit{PolicyBench}}, the first large-scale cross-system benchmark (US-China) evaluating policy comprehension, comprising 21K cases across a broad spectrum of policy areas, capturing the diversity and complexity of real-world governance. Following Bloom's taxonomy, the benchmark assesses three core capabilities: (1) \textbf{Memorization}: factual recall of policy knowledge, (2) \textbf{Understanding}: conceptual and contextual reasoning, and (3) \textbf{Application}: problem-solving in real-life policy scenarios. Building on this benchmark, we further propose \textbf{\textit{PolicyMoE}}, a domain-specialized Mixture-of-Experts (MoE) model with expert modules aligned to each cognitive level. The proposed models demonstrate stronger performance on application-oriented policy tasks than on memorization or conceptual understanding, and yields the highest accuracy on structured reasoning tasks. Our results reveal key limitations of current LLMs in policy understanding and suggest paths toward more reliable, policy-focused models. |
| title | PolicyLLM: Towards Excellent Comprehension of Public Policy for Large Language Models |
| topic | Computation and Language Computers and Society |
| url | https://arxiv.org/abs/2604.12995 |