FairLENS: Assessing Fairness in Law Enforcement Speech Recognition
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2024
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866910461612523520 |
|---|---|
| author | Wang, Yicheng Cusick, Mark Laila, Mohamed Puech, Kate Ji, Zhengping Hu, Xia Wilson, Michael Spitzer-Williams, Noah Wheeler, Bryan Ibrahim, Yasser |
| author_facet | Wang, Yicheng Cusick, Mark Laila, Mohamed Puech, Kate Ji, Zhengping Hu, Xia Wilson, Michael Spitzer-Williams, Noah Wheeler, Bryan Ibrahim, Yasser |
| contents | Automatic speech recognition (ASR) techniques have become powerful tools, enhancing efficiency in law enforcement scenarios. To ensure fairness for demographic groups in different acoustic environments, ASR engines must be tested across a variety of speakers in realistic settings. However, describing the fairness discrepancies between models with confidence remains a challenge. Meanwhile, most public ASR datasets are insufficient to perform a satisfying fairness evaluation. To address the limitations, we built FairLENS - a systematic fairness evaluation framework. We propose a novel and adaptable evaluation method to examine the fairness disparity between different models. We also collected a fairness evaluation dataset covering multiple scenarios and demographic dimensions. Leveraging this framework, we conducted fairness assessments on 1 open-source and 11 commercially available state-of-the-art ASR models. Our results reveal that certain models exhibit more biases than others, serving as a fairness guideline for users to make informed choices when selecting ASR models for a given real-world scenario. We further explored model biases towards specific demographic groups and observed that shifts in the acoustic domain can lead to the emergence of new biases. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2405_13166 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | FairLENS: Assessing Fairness in Law Enforcement Speech Recognition Wang, Yicheng Cusick, Mark Laila, Mohamed Puech, Kate Ji, Zhengping Hu, Xia Wilson, Michael Spitzer-Williams, Noah Wheeler, Bryan Ibrahim, Yasser Audio and Speech Processing Artificial Intelligence Computers and Society Automatic speech recognition (ASR) techniques have become powerful tools, enhancing efficiency in law enforcement scenarios. To ensure fairness for demographic groups in different acoustic environments, ASR engines must be tested across a variety of speakers in realistic settings. However, describing the fairness discrepancies between models with confidence remains a challenge. Meanwhile, most public ASR datasets are insufficient to perform a satisfying fairness evaluation. To address the limitations, we built FairLENS - a systematic fairness evaluation framework. We propose a novel and adaptable evaluation method to examine the fairness disparity between different models. We also collected a fairness evaluation dataset covering multiple scenarios and demographic dimensions. Leveraging this framework, we conducted fairness assessments on 1 open-source and 11 commercially available state-of-the-art ASR models. Our results reveal that certain models exhibit more biases than others, serving as a fairness guideline for users to make informed choices when selecting ASR models for a given real-world scenario. We further explored model biases towards specific demographic groups and observed that shifts in the acoustic domain can lead to the emergence of new biases. |
| title | FairLENS: Assessing Fairness in Law Enforcement Speech Recognition |
| topic | Audio and Speech Processing Artificial Intelligence Computers and Society |
| url | https://arxiv.org/abs/2405.13166 |