Efficiently Train ASR Models that Memorize Less and Perform Better with Per-core Clipping
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2024
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866911907647062016 |
|---|---|
| author | Wang, Lun Thakkar, Om Meng, Zhong Rafidi, Nicole Prabhavalkar, Rohit Narayanan, Arun |
| author_facet | Wang, Lun Thakkar, Om Meng, Zhong Rafidi, Nicole Prabhavalkar, Rohit Narayanan, Arun |
| contents | Gradient clipping plays a vital role in training large-scale automatic speech recognition (ASR) models. It is typically applied to minibatch gradients to prevent gradient explosion, and to the individual sample gradients to mitigate unintended memorization. This work systematically investigates the impact of a specific granularity of gradient clipping, namely per-core clip-ping (PCC), across training a wide range of ASR models. We empirically demonstrate that PCC can effectively mitigate unintended memorization in ASR models. Surprisingly, we find that PCC positively influences ASR performance metrics, leading to improved convergence rates and reduced word error rates. To avoid tuning the additional hyperparameter introduced by PCC, we further propose a novel variant, adaptive per-core clipping (APCC), for streamlined optimization. Our findings highlight the multifaceted benefits of PCC as a strategy for robust, privacy-forward ASR model training. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2406_02004 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Efficiently Train ASR Models that Memorize Less and Perform Better with Per-core Clipping Wang, Lun Thakkar, Om Meng, Zhong Rafidi, Nicole Prabhavalkar, Rohit Narayanan, Arun Cryptography and Security Computation and Language Sound Audio and Speech Processing Gradient clipping plays a vital role in training large-scale automatic speech recognition (ASR) models. It is typically applied to minibatch gradients to prevent gradient explosion, and to the individual sample gradients to mitigate unintended memorization. This work systematically investigates the impact of a specific granularity of gradient clipping, namely per-core clip-ping (PCC), across training a wide range of ASR models. We empirically demonstrate that PCC can effectively mitigate unintended memorization in ASR models. Surprisingly, we find that PCC positively influences ASR performance metrics, leading to improved convergence rates and reduced word error rates. To avoid tuning the additional hyperparameter introduced by PCC, we further propose a novel variant, adaptive per-core clipping (APCC), for streamlined optimization. Our findings highlight the multifaceted benefits of PCC as a strategy for robust, privacy-forward ASR model training. |
| title | Efficiently Train ASR Models that Memorize Less and Perform Better with Per-core Clipping |
| topic | Cryptography and Security Computation and Language Sound Audio and Speech Processing |
| url | https://arxiv.org/abs/2406.02004 |