CLEME2.0: Towards Interpretable Evaluation by Disentangling Edits for Grammatical Error Correction
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , , , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2024
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866915310756429824 |
|---|---|
| author | Ye, Jingheng Xu, Zishan Li, Yinghui Song, Linlin Zhou, Qingyu Zheng, Hai-Tao Shen, Ying Jiang, Wenhao Kim, Hong-Gee Liu, Ruitong Su, Xin Shan, Zifei |
| author_facet | Ye, Jingheng Xu, Zishan Li, Yinghui Song, Linlin Zhou, Qingyu Zheng, Hai-Tao Shen, Ying Jiang, Wenhao Kim, Hong-Gee Liu, Ruitong Su, Xin Shan, Zifei |
| contents | The paper focuses on the interpretability of Grammatical Error Correction (GEC) evaluation metrics, which received little attention in previous studies. To bridge the gap, we introduce **CLEME2.0**, a reference-based metric describing four fundamental aspects of GEC systems: hit-correction, wrong-correction, under-correction, and over-correction. They collectively contribute to exposing critical qualities and locating drawbacks of GEC systems. Evaluating systems by combining these aspects also leads to superior human consistency over other reference-based and reference-less metrics. Extensive experiments on two human judgment datasets and six reference datasets demonstrate the effectiveness and robustness of our method, achieving a new state-of-the-art result. Our codes are released at https://github.com/THUKElab/CLEME. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2407_00934 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | CLEME2.0: Towards Interpretable Evaluation by Disentangling Edits for Grammatical Error Correction Ye, Jingheng Xu, Zishan Li, Yinghui Song, Linlin Zhou, Qingyu Zheng, Hai-Tao Shen, Ying Jiang, Wenhao Kim, Hong-Gee Liu, Ruitong Su, Xin Shan, Zifei Computation and Language The paper focuses on the interpretability of Grammatical Error Correction (GEC) evaluation metrics, which received little attention in previous studies. To bridge the gap, we introduce **CLEME2.0**, a reference-based metric describing four fundamental aspects of GEC systems: hit-correction, wrong-correction, under-correction, and over-correction. They collectively contribute to exposing critical qualities and locating drawbacks of GEC systems. Evaluating systems by combining these aspects also leads to superior human consistency over other reference-based and reference-less metrics. Extensive experiments on two human judgment datasets and six reference datasets demonstrate the effectiveness and robustness of our method, achieving a new state-of-the-art result. Our codes are released at https://github.com/THUKElab/CLEME. |
| title | CLEME2.0: Towards Interpretable Evaluation by Disentangling Edits for Grammatical Error Correction |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2407.00934 |