ASR-FAIRBENCH: Measuring and Benchmarking Equity Across Speech Recognition Systems
Fuente:
arXiv
Saved in:
| Main Authors: | , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866913844062846976 |
|---|---|
| author | Rai, Anand Rahangdale, Satyam Anand, Utkarsh Mukherjee, Animesh |
| author_facet | Rai, Anand Rahangdale, Satyam Anand, Utkarsh Mukherjee, Animesh |
| contents | Automatic Speech Recognition (ASR) systems have become ubiquitous in everyday applications, yet significant disparities in performance across diverse demographic groups persist. In this work, we introduce the ASR-FAIRBENCH leaderboard which is designed to assess both the accuracy and equity of ASR models in real-time. Leveraging the Meta's Fair-Speech dataset, which captures diverse demographic characteristics, we employ a mixed-effects Poisson regression model to derive an overall fairness score. This score is integrated with traditional metrics like Word Error Rate (WER) to compute the Fairness Adjusted ASR Score (FAAS), providing a comprehensive evaluation framework. Our approach reveals significant performance disparities in SOTA ASR models across demographic groups and offers a benchmark to drive the development of more inclusive ASR technologies. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2505_11572 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | ASR-FAIRBENCH: Measuring and Benchmarking Equity Across Speech Recognition Systems Rai, Anand Rahangdale, Satyam Anand, Utkarsh Mukherjee, Animesh Sound Computation and Language Audio and Speech Processing Automatic Speech Recognition (ASR) systems have become ubiquitous in everyday applications, yet significant disparities in performance across diverse demographic groups persist. In this work, we introduce the ASR-FAIRBENCH leaderboard which is designed to assess both the accuracy and equity of ASR models in real-time. Leveraging the Meta's Fair-Speech dataset, which captures diverse demographic characteristics, we employ a mixed-effects Poisson regression model to derive an overall fairness score. This score is integrated with traditional metrics like Word Error Rate (WER) to compute the Fairness Adjusted ASR Score (FAAS), providing a comprehensive evaluation framework. Our approach reveals significant performance disparities in SOTA ASR models across demographic groups and offers a benchmark to drive the development of more inclusive ASR technologies. |
| title | ASR-FAIRBENCH: Measuring and Benchmarking Equity Across Speech Recognition Systems |
| topic | Sound Computation and Language Audio and Speech Processing |
| url | https://arxiv.org/abs/2505.11572 |