OYXOY: A Modern NLP Test Suite for Modern Greek
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2023
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866910308393549824 |
|---|---|
| author | Kogkalidis, Konstantinos Chatzikyriakidis, Stergios Giannikouri, Eirini Chrysovalantou Katsouli, Vassiliki Klironomou, Christina Koula, Christina Papadakis, Dimitris Pasparaki, Thelka Psaltaki, Erofili Sakellariou, Efthymia Soupiona, Hara |
| author_facet | Kogkalidis, Konstantinos Chatzikyriakidis, Stergios Giannikouri, Eirini Chrysovalantou Katsouli, Vassiliki Klironomou, Christina Koula, Christina Papadakis, Dimitris Pasparaki, Thelka Psaltaki, Erofili Sakellariou, Efthymia Soupiona, Hara |
| contents | This paper serves as a foundational step towards the development of a linguistically motivated and technically relevant evaluation suite for Greek NLP. We initiate this endeavor by introducing four expert-verified evaluation tasks, specifically targeted at natural language inference, word sense disambiguation (through example comparison or sense selection) and metaphor detection. More than language-adapted replicas of existing tasks, we contribute two innovations which will resonate with the broader resource and evaluation community. Firstly, our inference dataset is the first of its kind, marking not just \textit{one}, but rather \textit{all} possible inference labels, accounting for possible shifts due to e.g. ambiguity or polysemy. Secondly, we demonstrate a cost-efficient method to obtain datasets for under-resourced languages. Using ChatGPT as a language-neutral parser, we transform the Dictionary of Standard Modern Greek into a structured format, from which we derive the other three tasks through simple projections. Alongside each task, we conduct experiments using currently available state of the art machinery. Our experimental baselines affirm the challenging nature of our tasks and highlight the need for expedited progress in order for the Greek NLP ecosystem to keep pace with contemporary mainstream research. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2309_07009 |
| institution | arXiv |
| publishDate | 2023 |
| record_format | arxiv |
| spellingShingle | OYXOY: A Modern NLP Test Suite for Modern Greek Kogkalidis, Konstantinos Chatzikyriakidis, Stergios Giannikouri, Eirini Chrysovalantou Katsouli, Vassiliki Klironomou, Christina Koula, Christina Papadakis, Dimitris Pasparaki, Thelka Psaltaki, Erofili Sakellariou, Efthymia Soupiona, Hara Computation and Language This paper serves as a foundational step towards the development of a linguistically motivated and technically relevant evaluation suite for Greek NLP. We initiate this endeavor by introducing four expert-verified evaluation tasks, specifically targeted at natural language inference, word sense disambiguation (through example comparison or sense selection) and metaphor detection. More than language-adapted replicas of existing tasks, we contribute two innovations which will resonate with the broader resource and evaluation community. Firstly, our inference dataset is the first of its kind, marking not just \textit{one}, but rather \textit{all} possible inference labels, accounting for possible shifts due to e.g. ambiguity or polysemy. Secondly, we demonstrate a cost-efficient method to obtain datasets for under-resourced languages. Using ChatGPT as a language-neutral parser, we transform the Dictionary of Standard Modern Greek into a structured format, from which we derive the other three tasks through simple projections. Alongside each task, we conduct experiments using currently available state of the art machinery. Our experimental baselines affirm the challenging nature of our tasks and highlight the need for expedited progress in order for the Greek NLP ecosystem to keep pace with contemporary mainstream research. |
| title | OYXOY: A Modern NLP Test Suite for Modern Greek |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2309.07009 |