A Formal Framework for Fluency-based Multi-Reference Evaluation in Grammatical Error Correction
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866908581045993472 |
|---|---|
| author | Klinger, Eitan Huang, Zihao Nguyen, Tran Minh Park, Emma Jayeon Chen, Yige Gu, Yang Gao, Qingyu Liu, Siliang Qiu, Mengyang Park, Jungyeul |
| author_facet | Klinger, Eitan Huang, Zihao Nguyen, Tran Minh Park, Emma Jayeon Chen, Yige Gu, Yang Gao, Qingyu Liu, Siliang Qiu, Mengyang Park, Jungyeul |
| contents | Evaluating grammatical error correction requires metrics that reflect the diversity of valid human corrections rather than privileging a single reference. Existing frameworks, largely edit-based and English-centric, rely on rigid alignments between system and reference edits, limiting their applicability in multilingual and generative settings. This paper introduces a formal framework for \textit{fluency-based multi-reference evaluation}, framing $n$-gram similarity as an aggregation problem over multiple legitimate corrections. Within this formulation, we instantiate GLEU through four aggregation strategies--\textsc{select-best}, \textsc{simple-average}, \textsc{weighted-average}, and \textsc{merged-counts}--and analyze their properties of boundedness, monotonicity, and sensitivity to reference variation. Empirical results on Czech, Estonian, Ukrainian, and Chinese corpora show that these strategies capture complementary aspects of fluency and coverage. The framework unifies multi-reference evaluation into a principled, fluency-oriented approach that incorporates linguistic diversity without penalizing legitimate variation. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2510_06749 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | A Formal Framework for Fluency-based Multi-Reference Evaluation in Grammatical Error Correction Klinger, Eitan Huang, Zihao Nguyen, Tran Minh Park, Emma Jayeon Chen, Yige Gu, Yang Gao, Qingyu Liu, Siliang Qiu, Mengyang Park, Jungyeul Computation and Language Evaluating grammatical error correction requires metrics that reflect the diversity of valid human corrections rather than privileging a single reference. Existing frameworks, largely edit-based and English-centric, rely on rigid alignments between system and reference edits, limiting their applicability in multilingual and generative settings. This paper introduces a formal framework for \textit{fluency-based multi-reference evaluation}, framing $n$-gram similarity as an aggregation problem over multiple legitimate corrections. Within this formulation, we instantiate GLEU through four aggregation strategies--\textsc{select-best}, \textsc{simple-average}, \textsc{weighted-average}, and \textsc{merged-counts}--and analyze their properties of boundedness, monotonicity, and sensitivity to reference variation. Empirical results on Czech, Estonian, Ukrainian, and Chinese corpora show that these strategies capture complementary aspects of fluency and coverage. The framework unifies multi-reference evaluation into a principled, fluency-oriented approach that incorporates linguistic diversity without penalizing legitimate variation. |
| title | A Formal Framework for Fluency-based Multi-Reference Evaluation in Grammatical Error Correction |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2510.06749 |