When Wording Steers the Evaluation: Framing Bias in LLM judges
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2026
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866911386985037824 |
|---|---|
| author | Hwang, Yerin Lee, Dongryeol Kang, Taegwan Lee, Minwoo Jung, Kyomin |
| author_facet | Hwang, Yerin Lee, Dongryeol Kang, Taegwan Lee, Minwoo Jung, Kyomin |
| contents | Large language models (LLMs) are known to produce varying responses depending on prompt phrasing, indicating that subtle guidance in phrasing can steer their answers. However, the impact of this framing bias on LLM-based evaluation, where models are expected to make stable and impartial judgments, remains largely underexplored. Drawing inspiration from the framing effect in psychology, we systematically investigate how deliberate prompt framing skews model judgments across four high-stakes evaluation tasks. We design symmetric prompts using predicate-positive and predicate-negative constructions and demonstrate that such framing induces significant discrepancies in model outputs. Across 14 LLM judges, we observe clear susceptibility to framing, with model families showing distinct tendencies toward agreement or rejection. These findings suggest that framing bias is a structural property of current LLM-based evaluation systems, underscoring the need for framing-aware protocols. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2601_13537 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | When Wording Steers the Evaluation: Framing Bias in LLM judges Hwang, Yerin Lee, Dongryeol Kang, Taegwan Lee, Minwoo Jung, Kyomin Computation and Language Artificial Intelligence Large language models (LLMs) are known to produce varying responses depending on prompt phrasing, indicating that subtle guidance in phrasing can steer their answers. However, the impact of this framing bias on LLM-based evaluation, where models are expected to make stable and impartial judgments, remains largely underexplored. Drawing inspiration from the framing effect in psychology, we systematically investigate how deliberate prompt framing skews model judgments across four high-stakes evaluation tasks. We design symmetric prompts using predicate-positive and predicate-negative constructions and demonstrate that such framing induces significant discrepancies in model outputs. Across 14 LLM judges, we observe clear susceptibility to framing, with model families showing distinct tendencies toward agreement or rejection. These findings suggest that framing bias is a structural property of current LLM-based evaluation systems, underscoring the need for framing-aware protocols. |
| title | When Wording Steers the Evaluation: Framing Bias in LLM judges |
| topic | Computation and Language Artificial Intelligence |
| url | https://arxiv.org/abs/2601.13537 |