Saved in:
| Main Authors: | , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2026
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2605.16023 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866914598734528512 |
|---|---|
| author | Feldhus, Nils Baeumel, Tanja Golimblevskaia, Elena Wang, Qianli Nguyen, Van Bach Eidt, Aaron Louis Kahvecioglu, Selin Ebert, Christopher Samek, Wojciech Yang, Jing Schmitt, Vera Möller, Sebastian Ostermann, Simon |
| author_facet | Feldhus, Nils Baeumel, Tanja Golimblevskaia, Elena Wang, Qianli Nguyen, Van Bach Eidt, Aaron Louis Kahvecioglu, Selin Ebert, Christopher Samek, Wojciech Yang, Jing Schmitt, Vera Möller, Sebastian Ostermann, Simon |
| contents | LLM-as-a-judge has become the dominant paradigm for grading model outputs at scale, yet the same model assigns systematically different scores when its output format changes (e.g., a 1-5 rating vs. a True/False label). Existing diagnoses of these format-induced inconsistencies stop at the input-output level. Using Position-aware Edge Attribution Patching (PEAP), we causally investigate the internal mechanism in Gemma-3, Qwen2.5, and Llama-3. We find that judgments across structured understanding and open-ended preference tasks share a sparse, generalized Latent Evaluator sub-graph in the mid-to-late multi-layer perceptrons (MLPs); zero-ablating it collapses judgment while preserving world knowledge in architecturally modular models. By structurally decoupling abstract judging from output formatting, we provide a mechanistic account of format-induced inconsistency on the open-weight models we study: a continuous judgment signal computed in the shared trunk is mapped through fragile, format-specific terminal branches, enabling format-independent preference to be isolated downstream of the requested output format. Our findings imply that benchmark-level reliability comparisons across formats are partially measuring formatter geometry rather than evaluation quality. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2605_16023 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | Judge Circuits Feldhus, Nils Baeumel, Tanja Golimblevskaia, Elena Wang, Qianli Nguyen, Van Bach Eidt, Aaron Louis Kahvecioglu, Selin Ebert, Christopher Samek, Wojciech Yang, Jing Schmitt, Vera Möller, Sebastian Ostermann, Simon Computation and Language Machine Learning LLM-as-a-judge has become the dominant paradigm for grading model outputs at scale, yet the same model assigns systematically different scores when its output format changes (e.g., a 1-5 rating vs. a True/False label). Existing diagnoses of these format-induced inconsistencies stop at the input-output level. Using Position-aware Edge Attribution Patching (PEAP), we causally investigate the internal mechanism in Gemma-3, Qwen2.5, and Llama-3. We find that judgments across structured understanding and open-ended preference tasks share a sparse, generalized Latent Evaluator sub-graph in the mid-to-late multi-layer perceptrons (MLPs); zero-ablating it collapses judgment while preserving world knowledge in architecturally modular models. By structurally decoupling abstract judging from output formatting, we provide a mechanistic account of format-induced inconsistency on the open-weight models we study: a continuous judgment signal computed in the shared trunk is mapped through fragile, format-specific terminal branches, enabling format-independent preference to be isolated downstream of the requested output format. Our findings imply that benchmark-level reliability comparisons across formats are partially measuring formatter geometry rather than evaluation quality. |
| title | Judge Circuits |
| topic | Computation and Language Machine Learning |
| url | https://arxiv.org/abs/2605.16023 |