A Measure of the System Dependence of Automated Metrics
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2024
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866910766020427776 |
|---|---|
| author | von Däniken, Pius Deriu, Jan Cieliebak, Mark |
| author_facet | von Däniken, Pius Deriu, Jan Cieliebak, Mark |
| contents | Automated metrics for Machine Translation have made significant progress, with the goal of replacing expensive and time-consuming human evaluations. These metrics are typically assessed by their correlation with human judgments, which captures the monotonic relationship between human and metric scores. However, we argue that it is equally important to ensure that metrics treat all systems fairly and consistently. In this paper, we introduce a method to evaluate this aspect. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2412_03152 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | A Measure of the System Dependence of Automated Metrics von Däniken, Pius Deriu, Jan Cieliebak, Mark Computation and Language Artificial Intelligence Automated metrics for Machine Translation have made significant progress, with the goal of replacing expensive and time-consuming human evaluations. These metrics are typically assessed by their correlation with human judgments, which captures the monotonic relationship between human and metric scores. However, we argue that it is equally important to ensure that metrics treat all systems fairly and consistently. In this paper, we introduce a method to evaluate this aspect. |
| title | A Measure of the System Dependence of Automated Metrics |
| topic | Computation and Language Artificial Intelligence |
| url | https://arxiv.org/abs/2412.03152 |