WUGNECTIVES: Novel Entity Inferences of Language Models from Discourse Connectives
Fuente:
arXiv
Saved in:
| Main Authors: | , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866918318441496576 |
|---|---|
| author | Brubaker, Daniel Sheffield, William Li, Junyi Jessy Misra, Kanishka |
| author_facet | Brubaker, Daniel Sheffield, William Li, Junyi Jessy Misra, Kanishka |
| contents | The role of world knowledge has been particularly crucial to predict the discourse connective that marks the discourse relation between two arguments, with language models (LMs) being generally successful at this task. We flip this premise in our work, and instead study the inverse problem of understanding whether discourse connectives can inform LMs about the world. To this end, we present WUGNECTIVES, a dataset of 8,880 stimuli that evaluates LMs' inferences about novel entities in contexts where connectives link the entities to particular attributes. On investigating 17 different LMs at various scales, and training regimens, we found that tuning an LM to show reasoning behavior yields noteworthy improvements on most connectives. At the same time, there was a large variation in LMs' overall performance across connective type, with all models systematically struggling on connectives that express a concessive meaning. Our findings pave the way for more nuanced investigations into the functional role of language cues as captured by LMs. We release WUGNECTIVES at https://github.com/kanishkamisra/wugnectives |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2510_09556 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | WUGNECTIVES: Novel Entity Inferences of Language Models from Discourse Connectives Brubaker, Daniel Sheffield, William Li, Junyi Jessy Misra, Kanishka Computation and Language The role of world knowledge has been particularly crucial to predict the discourse connective that marks the discourse relation between two arguments, with language models (LMs) being generally successful at this task. We flip this premise in our work, and instead study the inverse problem of understanding whether discourse connectives can inform LMs about the world. To this end, we present WUGNECTIVES, a dataset of 8,880 stimuli that evaluates LMs' inferences about novel entities in contexts where connectives link the entities to particular attributes. On investigating 17 different LMs at various scales, and training regimens, we found that tuning an LM to show reasoning behavior yields noteworthy improvements on most connectives. At the same time, there was a large variation in LMs' overall performance across connective type, with all models systematically struggling on connectives that express a concessive meaning. Our findings pave the way for more nuanced investigations into the functional role of language cues as captured by LMs. We release WUGNECTIVES at https://github.com/kanishkamisra/wugnectives |
| title | WUGNECTIVES: Novel Entity Inferences of Language Models from Discourse Connectives |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2510.09556 |