Finding Task-specific Subnetworks in Multi-task Spoken Language Understanding Model
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866910492055830528 |
|---|---|
| author | Futami, Hayato Arora, Siddhant Kashiwagi, Yosuke Tsunoo, Emiru Watanabe, Shinji |
| author_facet | Futami, Hayato Arora, Siddhant Kashiwagi, Yosuke Tsunoo, Emiru Watanabe, Shinji |
| contents | Recently, multi-task spoken language understanding (SLU) models have emerged, designed to address various speech processing tasks. However, these models often rely on a large number of parameters. Also, they often encounter difficulties in adapting to new data for a specific task without experiencing catastrophic forgetting of previously trained tasks. In this study, we propose finding task-specific subnetworks within a multi-task SLU model via neural network pruning. In addition to model compression, we expect that the forgetting of previously trained tasks can be mitigated by updating only a task-specific subnetwork. We conduct experiments on top of the state-of-the-art multi-task SLU model ``UniverSLU'', trained for several tasks such as emotion recognition (ER), intent classification (IC), and automatic speech recognition (ASR). We show that pruned models were successful in adapting to additional ASR or IC data with minimal performance degradation on previously trained tasks. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2406_12317 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Finding Task-specific Subnetworks in Multi-task Spoken Language Understanding Model Futami, Hayato Arora, Siddhant Kashiwagi, Yosuke Tsunoo, Emiru Watanabe, Shinji Computation and Language Audio and Speech Processing Recently, multi-task spoken language understanding (SLU) models have emerged, designed to address various speech processing tasks. However, these models often rely on a large number of parameters. Also, they often encounter difficulties in adapting to new data for a specific task without experiencing catastrophic forgetting of previously trained tasks. In this study, we propose finding task-specific subnetworks within a multi-task SLU model via neural network pruning. In addition to model compression, we expect that the forgetting of previously trained tasks can be mitigated by updating only a task-specific subnetwork. We conduct experiments on top of the state-of-the-art multi-task SLU model ``UniverSLU'', trained for several tasks such as emotion recognition (ER), intent classification (IC), and automatic speech recognition (ASR). We show that pruned models were successful in adapting to additional ASR or IC data with minimal performance degradation on previously trained tasks. |
| title | Finding Task-specific Subnetworks in Multi-task Spoken Language Understanding Model |
| topic | Computation and Language Audio and Speech Processing |
| url | https://arxiv.org/abs/2406.12317 |