How to Make the Most of LLMs' Grammatical Knowledge for Acceptability Judgments
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866912222895144960 |
|---|---|
| author | Ide, Yusuke Nishida, Yuto Vasselli, Justin Oba, Miyu Sakai, Yusuke Kamigaito, Hidetaka Watanabe, Taro |
| author_facet | Ide, Yusuke Nishida, Yuto Vasselli, Justin Oba, Miyu Sakai, Yusuke Kamigaito, Hidetaka Watanabe, Taro |
| contents | The grammatical knowledge of language models (LMs) is often measured using a benchmark of linguistic minimal pairs, where the LMs are presented with a pair of acceptable and unacceptable sentences and required to judge which is more acceptable. Conventional approaches directly compare sentence probabilities assigned by LMs, but recent large language models (LLMs) are trained to perform tasks via prompting, and thus, the raw probabilities they assign may not fully reflect their grammatical knowledge. In this study, we attempt to derive more accurate acceptability judgments from LLMs using prompts and templates. Through extensive experiments in English and Chinese, we compare nine judgment methods and find two of them, a probability readout method -- in-template LP and a prompt-based method -- Yes/No probability computing, achieve higher accuracy than the conventional ones. Our analysis reveals that these methods excel in different linguistic phenomena, suggesting they access different aspects of LLMs' knowledge. We also find that ensembling the two methods outperforms single methods. Consequently, we recommend these techniques, either individually or ensembled, as more effective alternatives to conventional approaches for assessing grammatical knowledge in LLMs. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2408_09639 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | How to Make the Most of LLMs' Grammatical Knowledge for Acceptability Judgments Ide, Yusuke Nishida, Yuto Vasselli, Justin Oba, Miyu Sakai, Yusuke Kamigaito, Hidetaka Watanabe, Taro Computation and Language Artificial Intelligence The grammatical knowledge of language models (LMs) is often measured using a benchmark of linguistic minimal pairs, where the LMs are presented with a pair of acceptable and unacceptable sentences and required to judge which is more acceptable. Conventional approaches directly compare sentence probabilities assigned by LMs, but recent large language models (LLMs) are trained to perform tasks via prompting, and thus, the raw probabilities they assign may not fully reflect their grammatical knowledge. In this study, we attempt to derive more accurate acceptability judgments from LLMs using prompts and templates. Through extensive experiments in English and Chinese, we compare nine judgment methods and find two of them, a probability readout method -- in-template LP and a prompt-based method -- Yes/No probability computing, achieve higher accuracy than the conventional ones. Our analysis reveals that these methods excel in different linguistic phenomena, suggesting they access different aspects of LLMs' knowledge. We also find that ensembling the two methods outperforms single methods. Consequently, we recommend these techniques, either individually or ensembled, as more effective alternatives to conventional approaches for assessing grammatical knowledge in LLMs. |
| title | How to Make the Most of LLMs' Grammatical Knowledge for Acceptability Judgments |
| topic | Computation and Language Artificial Intelligence |
| url | https://arxiv.org/abs/2408.09639 |