As Biased as You Measure: Methodological Pitfalls of Bias Evaluations in Speaker Verification Research
Fuente:
arXiv
Saved in:
| Main Authors: | , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866913479032569856 |
|---|---|
| author | Hutiri, Wiebke Patel, Tanvina Ding, Aaron Yi Scharenborg, Odette |
| author_facet | Hutiri, Wiebke Patel, Tanvina Ding, Aaron Yi Scharenborg, Odette |
| contents | Detecting and mitigating bias in speaker verification systems is important, as datasets, processing choices and algorithms can lead to performance differences that systematically favour some groups of people while disadvantaging others. Prior studies have thus measured performance differences across groups to evaluate bias. However, when comparing results across studies, it becomes apparent that they draw contradictory conclusions, hindering progress in this area. In this paper we investigate how measurement impacts the outcomes of bias evaluations. We show empirically that bias evaluations are strongly influenced by base metrics that measure performance, by the choice of ratio or difference-based bias measure, and by the aggregation of bias measures into meta-measures. Based on our findings, we recommend the use of ratio-based bias measures, in particular when the values of base metrics are small, or when base metrics with different orders of magnitude need to be compared. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2408_13614 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | As Biased as You Measure: Methodological Pitfalls of Bias Evaluations in Speaker Verification Research Hutiri, Wiebke Patel, Tanvina Ding, Aaron Yi Scharenborg, Odette Audio and Speech Processing Computers and Society Detecting and mitigating bias in speaker verification systems is important, as datasets, processing choices and algorithms can lead to performance differences that systematically favour some groups of people while disadvantaging others. Prior studies have thus measured performance differences across groups to evaluate bias. However, when comparing results across studies, it becomes apparent that they draw contradictory conclusions, hindering progress in this area. In this paper we investigate how measurement impacts the outcomes of bias evaluations. We show empirically that bias evaluations are strongly influenced by base metrics that measure performance, by the choice of ratio or difference-based bias measure, and by the aggregation of bias measures into meta-measures. Based on our findings, we recommend the use of ratio-based bias measures, in particular when the values of base metrics are small, or when base metrics with different orders of magnitude need to be compared. |
| title | As Biased as You Measure: Methodological Pitfalls of Bias Evaluations in Speaker Verification Research |
| topic | Audio and Speech Processing Computers and Society |
| url | https://arxiv.org/abs/2408.13614 |