Z-Scores: A Metric for Linguistically Assessing Disfluency Removal
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866916967747682304 |
|---|---|
| author | Teleki, Maria Janjur, Sai Liu, Haoran Grabner, Oliver Verma, Ketan Docog, Thomas Dong, Xiangjue Shi, Lingfeng Wang, Cong Birkelbach, Stephanie Kim, Jason Zhang, Yin Caverlee, James |
| author_facet | Teleki, Maria Janjur, Sai Liu, Haoran Grabner, Oliver Verma, Ketan Docog, Thomas Dong, Xiangjue Shi, Lingfeng Wang, Cong Birkelbach, Stephanie Kim, Jason Zhang, Yin Caverlee, James |
| contents | Evaluating disfluency removal in speech requires more than aggregate token-level scores. Traditional word-based metrics such as precision, recall, and F1 (E-Scores) capture overall performance but cannot reveal why models succeed or fail. We introduce Z-Scores, a span-level linguistically-grounded evaluation metric that categorizes system behavior across distinct disfluency types (EDITED, INTJ, PRN). Our deterministic alignment module enables robust mapping between generated text and disfluent transcripts, allowing Z-Scores to expose systematic weaknesses that word-level metrics obscure. By providing category-specific diagnostics, Z-Scores enable researchers to identify model failure modes and design targeted interventions -- such as tailored prompts or data augmentation -- yielding measurable performance improvements. A case study with LLMs shows that Z-Scores uncover challenges with INTJ and PRN disfluencies hidden in aggregate F1, directly informing model refinement strategies. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2509_20319 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Z-Scores: A Metric for Linguistically Assessing Disfluency Removal Teleki, Maria Janjur, Sai Liu, Haoran Grabner, Oliver Verma, Ketan Docog, Thomas Dong, Xiangjue Shi, Lingfeng Wang, Cong Birkelbach, Stephanie Kim, Jason Zhang, Yin Caverlee, James Computation and Language Artificial Intelligence Audio and Speech Processing Evaluating disfluency removal in speech requires more than aggregate token-level scores. Traditional word-based metrics such as precision, recall, and F1 (E-Scores) capture overall performance but cannot reveal why models succeed or fail. We introduce Z-Scores, a span-level linguistically-grounded evaluation metric that categorizes system behavior across distinct disfluency types (EDITED, INTJ, PRN). Our deterministic alignment module enables robust mapping between generated text and disfluent transcripts, allowing Z-Scores to expose systematic weaknesses that word-level metrics obscure. By providing category-specific diagnostics, Z-Scores enable researchers to identify model failure modes and design targeted interventions -- such as tailored prompts or data augmentation -- yielding measurable performance improvements. A case study with LLMs shows that Z-Scores uncover challenges with INTJ and PRN disfluencies hidden in aggregate F1, directly informing model refinement strategies. |
| title | Z-Scores: A Metric for Linguistically Assessing Disfluency Removal |
| topic | Computation and Language Artificial Intelligence Audio and Speech Processing |
| url | https://arxiv.org/abs/2509.20319 |