TextEE: Benchmark, Reevaluation, Reflections, and Future Challenges in Event Extraction
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2023
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866909217850392576 |
|---|---|
| author | Huang, Kuan-Hao Hsu, I-Hung Parekh, Tanmay Xie, Zhiyu Zhang, Zixuan Natarajan, Premkumar Chang, Kai-Wei Peng, Nanyun Ji, Heng |
| author_facet | Huang, Kuan-Hao Hsu, I-Hung Parekh, Tanmay Xie, Zhiyu Zhang, Zixuan Natarajan, Premkumar Chang, Kai-Wei Peng, Nanyun Ji, Heng |
| contents | Event extraction has gained considerable interest due to its wide-ranging applications. However, recent studies draw attention to evaluation issues, suggesting that reported scores may not accurately reflect the true performance. In this work, we identify and address evaluation challenges, including inconsistency due to varying data assumptions or preprocessing steps, the insufficiency of current evaluation frameworks that may introduce dataset or data split bias, and the low reproducibility of some previous approaches. To address these challenges, we present TextEE, a standardized, fair, and reproducible benchmark for event extraction. TextEE comprises standardized data preprocessing scripts and splits for 16 datasets spanning eight diverse domains and includes 14 recent methodologies, conducting a comprehensive benchmark reevaluation. We also evaluate five varied large language models on our TextEE benchmark and demonstrate how they struggle to achieve satisfactory performance. Inspired by our reevaluation results and findings, we discuss the role of event extraction in the current NLP era, as well as future challenges and insights derived from TextEE. We believe TextEE, the first standardized comprehensive benchmarking tool, will significantly facilitate future event extraction research. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2311_09562 |
| institution | arXiv |
| publishDate | 2023 |
| record_format | arxiv |
| spellingShingle | TextEE: Benchmark, Reevaluation, Reflections, and Future Challenges in Event Extraction Huang, Kuan-Hao Hsu, I-Hung Parekh, Tanmay Xie, Zhiyu Zhang, Zixuan Natarajan, Premkumar Chang, Kai-Wei Peng, Nanyun Ji, Heng Computation and Language Event extraction has gained considerable interest due to its wide-ranging applications. However, recent studies draw attention to evaluation issues, suggesting that reported scores may not accurately reflect the true performance. In this work, we identify and address evaluation challenges, including inconsistency due to varying data assumptions or preprocessing steps, the insufficiency of current evaluation frameworks that may introduce dataset or data split bias, and the low reproducibility of some previous approaches. To address these challenges, we present TextEE, a standardized, fair, and reproducible benchmark for event extraction. TextEE comprises standardized data preprocessing scripts and splits for 16 datasets spanning eight diverse domains and includes 14 recent methodologies, conducting a comprehensive benchmark reevaluation. We also evaluate five varied large language models on our TextEE benchmark and demonstrate how they struggle to achieve satisfactory performance. Inspired by our reevaluation results and findings, we discuss the role of event extraction in the current NLP era, as well as future challenges and insights derived from TextEE. We believe TextEE, the first standardized comprehensive benchmarking tool, will significantly facilitate future event extraction research. |
| title | TextEE: Benchmark, Reevaluation, Reflections, and Future Challenges in Event Extraction |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2311.09562 |