BMX: Boosting Natural Language Generation Metrics with Explainability
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2022
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866929246129094656 |
|---|---|
| author | Leiter, Christoph Nguyen, Hoa Eger, Steffen |
| author_facet | Leiter, Christoph Nguyen, Hoa Eger, Steffen |
| contents | State-of-the-art natural language generation evaluation metrics are based on black-box language models. Hence, recent works consider their explainability with the goals of better understandability for humans and better metric analysis, including failure cases. In contrast, our proposed method BMX: Boosting Natural Language Generation Metrics with explainability explicitly leverages explanations to boost the metrics' performance. In particular, we perceive feature importance explanations as word-level scores, which we convert, via power means, into a segment-level score. We then combine this segment-level score with the original metric to obtain a better metric. Our tests show improvements for multiple metrics across MT and summarization datasets. While improvements in machine translation are small, they are strong for summarization. Notably, BMX with the LIME explainer and preselected parameters achieves an average improvement of 0.087 points in Spearman correlation on the system-level evaluation of SummEval. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2212_10469 |
| institution | arXiv |
| publishDate | 2022 |
| record_format | arxiv |
| spellingShingle | BMX: Boosting Natural Language Generation Metrics with Explainability Leiter, Christoph Nguyen, Hoa Eger, Steffen Computation and Language State-of-the-art natural language generation evaluation metrics are based on black-box language models. Hence, recent works consider their explainability with the goals of better understandability for humans and better metric analysis, including failure cases. In contrast, our proposed method BMX: Boosting Natural Language Generation Metrics with explainability explicitly leverages explanations to boost the metrics' performance. In particular, we perceive feature importance explanations as word-level scores, which we convert, via power means, into a segment-level score. We then combine this segment-level score with the original metric to obtain a better metric. Our tests show improvements for multiple metrics across MT and summarization datasets. While improvements in machine translation are small, they are strong for summarization. Notably, BMX with the LIME explainer and preselected parameters achieves an average improvement of 0.087 points in Spearman correlation on the system-level evaluation of SummEval. |
| title | BMX: Boosting Natural Language Generation Metrics with Explainability |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2212.10469 |