Gespeichert in:
| Hauptverfasser: | , , , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2026
|
| Schlagworte: | |
| Online-Zugang: | https://arxiv.org/abs/2603.09993 |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866908877528760320 |
|---|---|
| author | Chun, Jon Sussman, Hannah Mangine, Adrian Kocaman, Murathan Sidorko, Kirill Koirala, Abhigya McCloud, Andre Eisenbeis, Gwen Akanwe, Wisdom Gassama, Moustapha Chirinos, Eliezer Gonzalez Enright, Anne-Duncan Dunson, Peter Ng, Tiffanie von Rosenstiel, Anna Idowu, Godwin |
| author_facet | Chun, Jon Sussman, Hannah Mangine, Adrian Kocaman, Murathan Sidorko, Kirill Koirala, Abhigya McCloud, Andre Eisenbeis, Gwen Akanwe, Wisdom Gassama, Moustapha Chirinos, Eliezer Gonzalez Enright, Anne-Duncan Dunson, Peter Ng, Tiffanie von Rosenstiel, Anna Idowu, Godwin |
| contents | Pragmatic reasoning, inferring intended meaning beyond literal semantics, underpins everyday communication yet remains difficult for large language models. We present the Contextual Emotional Inference (CEI) Benchmark: 300 human-validated scenarios for evaluating how well LLMs disambiguate pragmatically complex utterances. Each scenario pairs a situational context and speaker-listener roles (with explicit power relations) against an ambiguous utterance. The dataset covers five pragmatic subtypes (sarcasm/irony, mixed signals, strategic politeness, passive aggression, deflection/misdirection) drawn from workplace, family, social, and service settings, with three power configurations (peer, higher-to-lower, lower-to-higher). Three trained annotators independently labeled every scenario. Inter-annotator agreement (Fleiss' kappa = 0.06-0.25 by subtype) is low but expected: pragmatic inference admits multiple valid readings, and the disagreement itself is informative. We describe our annotation methodology, including a 4-level quality control pipeline that combines automated statistical checks with expert adjudication. CEI is released under CC-BY-4.0. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2603_09993 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | CEI: A Benchmark for Evaluating Pragmatic Reasoning in Language Models Chun, Jon Sussman, Hannah Mangine, Adrian Kocaman, Murathan Sidorko, Kirill Koirala, Abhigya McCloud, Andre Eisenbeis, Gwen Akanwe, Wisdom Gassama, Moustapha Chirinos, Eliezer Gonzalez Enright, Anne-Duncan Dunson, Peter Ng, Tiffanie von Rosenstiel, Anna Idowu, Godwin Computation and Language Artificial Intelligence Pragmatic reasoning, inferring intended meaning beyond literal semantics, underpins everyday communication yet remains difficult for large language models. We present the Contextual Emotional Inference (CEI) Benchmark: 300 human-validated scenarios for evaluating how well LLMs disambiguate pragmatically complex utterances. Each scenario pairs a situational context and speaker-listener roles (with explicit power relations) against an ambiguous utterance. The dataset covers five pragmatic subtypes (sarcasm/irony, mixed signals, strategic politeness, passive aggression, deflection/misdirection) drawn from workplace, family, social, and service settings, with three power configurations (peer, higher-to-lower, lower-to-higher). Three trained annotators independently labeled every scenario. Inter-annotator agreement (Fleiss' kappa = 0.06-0.25 by subtype) is low but expected: pragmatic inference admits multiple valid readings, and the disagreement itself is informative. We describe our annotation methodology, including a 4-level quality control pipeline that combines automated statistical checks with expert adjudication. CEI is released under CC-BY-4.0. |
| title | CEI: A Benchmark for Evaluating Pragmatic Reasoning in Language Models |
| topic | Computation and Language Artificial Intelligence |
| url | https://arxiv.org/abs/2603.09993 |