Better Benchmarking LLMs for Zero-Shot Dependency Parsing
Fuente:
arXiv
Saved in:
| Main Authors: | , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866916635154055168 |
|---|---|
| author | Ezquerro, Ana Gómez-Rodríguez, Carlos Vilares, David |
| author_facet | Ezquerro, Ana Gómez-Rodríguez, Carlos Vilares, David |
| contents | While LLMs excel in zero-shot tasks, their performance in linguistic challenges like syntactic parsing has been less scrutinized. This paper studies state-of-the-art open-weight LLMs on the task by comparing them to baselines that do not have access to the input sentence, including baselines that have not been used in this context such as random projective trees or optimal linear arrangements. The results show that most of the tested LLMs cannot outperform the best uninformed baselines, with only the newest and largest versions of LLaMA doing so for most languages, and still achieving rather low performance. Thus, accurate zero-shot syntactic parsing is not forthcoming with open LLMs. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2502_20866 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Better Benchmarking LLMs for Zero-Shot Dependency Parsing Ezquerro, Ana Gómez-Rodríguez, Carlos Vilares, David Computation and Language While LLMs excel in zero-shot tasks, their performance in linguistic challenges like syntactic parsing has been less scrutinized. This paper studies state-of-the-art open-weight LLMs on the task by comparing them to baselines that do not have access to the input sentence, including baselines that have not been used in this context such as random projective trees or optimal linear arrangements. The results show that most of the tested LLMs cannot outperform the best uninformed baselines, with only the newest and largest versions of LLaMA doing so for most languages, and still achieving rather low performance. Thus, accurate zero-shot syntactic parsing is not forthcoming with open LLMs. |
| title | Better Benchmarking LLMs for Zero-Shot Dependency Parsing |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2502.20866 |