Trick or Neat: Adversarial Ambiguity and Language Model Evaluation
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2025
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866912409049890816 |
|---|---|
| author | Karamolegkou, Antonia Eberle, Oliver Rust, Phillip Kauf, Carina Søgaard, Anders |
| author_facet | Karamolegkou, Antonia Eberle, Oliver Rust, Phillip Kauf, Carina Søgaard, Anders |
| contents | Detecting ambiguity is important for language understanding, including uncertainty estimation, humour detection, and processing garden path sentences. We assess language models' sensitivity to ambiguity by introducing an adversarial ambiguity dataset that includes syntactic, lexical, and phonological ambiguities along with adversarial variations (e.g., word-order changes, synonym replacements, and random-based alterations). Our findings show that direct prompting fails to robustly identify ambiguity, while linear probes trained on model representations can decode ambiguity with high accuracy, sometimes exceeding 90\%. Our results offer insights into the prompting paradigm and how language models encode ambiguity at different layers. We release both our code and data: https://github.com/coastalcph/lm_ambiguity. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2506_01205 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Trick or Neat: Adversarial Ambiguity and Language Model Evaluation Karamolegkou, Antonia Eberle, Oliver Rust, Phillip Kauf, Carina Søgaard, Anders Computation and Language Detecting ambiguity is important for language understanding, including uncertainty estimation, humour detection, and processing garden path sentences. We assess language models' sensitivity to ambiguity by introducing an adversarial ambiguity dataset that includes syntactic, lexical, and phonological ambiguities along with adversarial variations (e.g., word-order changes, synonym replacements, and random-based alterations). Our findings show that direct prompting fails to robustly identify ambiguity, while linear probes trained on model representations can decode ambiguity with high accuracy, sometimes exceeding 90\%. Our results offer insights into the prompting paradigm and how language models encode ambiguity at different layers. We release both our code and data: https://github.com/coastalcph/lm_ambiguity. |
| title | Trick or Neat: Adversarial Ambiguity and Language Model Evaluation |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2506.01205 |