EXCGEC: A Benchmark for Edit-Wise Explainable Chinese Grammatical Error Correction
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866910829611319296 |
|---|---|
| author | Ye, Jingheng Qin, Shang Li, Yinghui Cheng, Xuxin Qin, Libo Zheng, Hai-Tao Shen, Ying Xing, Peng Xu, Zishan Cheng, Guo Jiang, Wenhao |
| author_facet | Ye, Jingheng Qin, Shang Li, Yinghui Cheng, Xuxin Qin, Libo Zheng, Hai-Tao Shen, Ying Xing, Peng Xu, Zishan Cheng, Guo Jiang, Wenhao |
| contents | Existing studies explore the explainability of Grammatical Error Correction (GEC) in a limited scenario, where they ignore the interaction between corrections and explanations and have not established a corresponding comprehensive benchmark. To bridge the gap, this paper first introduces the task of EXplainable GEC (EXGEC), which focuses on the integral role of correction and explanation tasks. To facilitate the task, we propose EXCGEC, a tailored benchmark for Chinese EXGEC consisting of 8,216 explanation-augmented samples featuring the design of hybrid edit-wise explanations. We then benchmark several series of LLMs in multi-task learning settings, including post-explaining and pre-explaining. To promote the development of the task, we also build a comprehensive evaluation suite by leveraging existing automatic metrics and conducting human evaluation experiments to demonstrate the human consistency of the automatic metrics for free-text explanations. Our experiments reveal the effectiveness of evaluating free-text explanations using traditional metrics like METEOR and ROUGE, and the inferior performance of multi-task models compared to the pipeline solution, indicating its challenges to establish positive effects in learning both tasks. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2407_00924 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | EXCGEC: A Benchmark for Edit-Wise Explainable Chinese Grammatical Error Correction Ye, Jingheng Qin, Shang Li, Yinghui Cheng, Xuxin Qin, Libo Zheng, Hai-Tao Shen, Ying Xing, Peng Xu, Zishan Cheng, Guo Jiang, Wenhao Computation and Language Existing studies explore the explainability of Grammatical Error Correction (GEC) in a limited scenario, where they ignore the interaction between corrections and explanations and have not established a corresponding comprehensive benchmark. To bridge the gap, this paper first introduces the task of EXplainable GEC (EXGEC), which focuses on the integral role of correction and explanation tasks. To facilitate the task, we propose EXCGEC, a tailored benchmark for Chinese EXGEC consisting of 8,216 explanation-augmented samples featuring the design of hybrid edit-wise explanations. We then benchmark several series of LLMs in multi-task learning settings, including post-explaining and pre-explaining. To promote the development of the task, we also build a comprehensive evaluation suite by leveraging existing automatic metrics and conducting human evaluation experiments to demonstrate the human consistency of the automatic metrics for free-text explanations. Our experiments reveal the effectiveness of evaluating free-text explanations using traditional metrics like METEOR and ROUGE, and the inferior performance of multi-task models compared to the pipeline solution, indicating its challenges to establish positive effects in learning both tasks. |
| title | EXCGEC: A Benchmark for Edit-Wise Explainable Chinese Grammatical Error Correction |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2407.00924 |