Saved in:
| Main Authors: | , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2410.16011 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866913557694644224 |
|---|---|
| author | Xu, Xi Xu, Wenda Ouyang, Siqi Li, Lei |
| author_facet | Xu, Xi Xu, Wenda Ouyang, Siqi Li, Lei |
| contents | Simultaneous speech translation (SimulST) systems must balance translation quality with response time, making latency measurement crucial for evaluating their real-world performance. However, there has been a longstanding belief that current metrics yield unrealistically high latency measurements in unsegmented streaming settings. In this paper, we investigate this phenomenon, revealing its root cause in a fundamental misconception underlying existing latency evaluation approaches. We demonstrate that this issue affects not only streaming but also segment-level latency evaluation across different metrics. Furthermore, we propose a modification to correctly measure computation-aware latency for SimulST systems, addressing the limitations present in existing metrics. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2410_16011 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | CA*: Addressing Evaluation Pitfalls in Computation-Aware Latency for Simultaneous Speech Translation Xu, Xi Xu, Wenda Ouyang, Siqi Li, Lei Computation and Language Artificial Intelligence Simultaneous speech translation (SimulST) systems must balance translation quality with response time, making latency measurement crucial for evaluating their real-world performance. However, there has been a longstanding belief that current metrics yield unrealistically high latency measurements in unsegmented streaming settings. In this paper, we investigate this phenomenon, revealing its root cause in a fundamental misconception underlying existing latency evaluation approaches. We demonstrate that this issue affects not only streaming but also segment-level latency evaluation across different metrics. Furthermore, we propose a modification to correctly measure computation-aware latency for SimulST systems, addressing the limitations present in existing metrics. |
| title | CA*: Addressing Evaluation Pitfalls in Computation-Aware Latency for Simultaneous Speech Translation |
| topic | Computation and Language Artificial Intelligence |
| url | https://arxiv.org/abs/2410.16011 |