Towards Realistic Few-Shot Relation Extraction: A New Meta Dataset and Evaluation
Fuente:
arXiv
Saved in:
| Main Authors: | , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866913302343319552 |
|---|---|
| author | Alam, Fahmida Islam, Md Asiful Vacareanu, Robert Surdeanu, Mihai |
| author_facet | Alam, Fahmida Islam, Md Asiful Vacareanu, Robert Surdeanu, Mihai |
| contents | We introduce a meta dataset for few-shot relation extraction, which includes two datasets derived from existing supervised relation extraction datasets NYT29 (Takanobu et al., 2019; Nayak and Ng, 2020) and WIKIDATA (Sorokin and Gurevych, 2017) as well as a few-shot form of the TACRED dataset (Sabo et al., 2021). Importantly, all these few-shot datasets were generated under realistic assumptions such as: the test relations are different from any relations a model might have seen before, limited training data, and a preponderance of candidate relation mentions that do not correspond to any of the relations of interest. Using this large resource, we conduct a comprehensive evaluation of six recent few-shot relation extraction methods, and observe that no method comes out as a clear winner. Further, the overall performance on this task is low, indicating substantial need for future research. We release all versions of the data, i.e., both supervised and few-shot, for future research. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2404_04445 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Towards Realistic Few-Shot Relation Extraction: A New Meta Dataset and Evaluation Alam, Fahmida Islam, Md Asiful Vacareanu, Robert Surdeanu, Mihai Computation and Language Information Retrieval We introduce a meta dataset for few-shot relation extraction, which includes two datasets derived from existing supervised relation extraction datasets NYT29 (Takanobu et al., 2019; Nayak and Ng, 2020) and WIKIDATA (Sorokin and Gurevych, 2017) as well as a few-shot form of the TACRED dataset (Sabo et al., 2021). Importantly, all these few-shot datasets were generated under realistic assumptions such as: the test relations are different from any relations a model might have seen before, limited training data, and a preponderance of candidate relation mentions that do not correspond to any of the relations of interest. Using this large resource, we conduct a comprehensive evaluation of six recent few-shot relation extraction methods, and observe that no method comes out as a clear winner. Further, the overall performance on this task is low, indicating substantial need for future research. We release all versions of the data, i.e., both supervised and few-shot, for future research. |
| title | Towards Realistic Few-Shot Relation Extraction: A New Meta Dataset and Evaluation |
| topic | Computation and Language Information Retrieval |
| url | https://arxiv.org/abs/2404.04445 |