Saved in:
| Main Authors: | , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2406.14459 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866916830460772352 |
|---|---|
| author | Han, Shijie Zhang, Zhenyu Simion, Andrei Arsene |
| author_facet | Han, Shijie Zhang, Zhenyu Simion, Andrei Arsene |
| contents | Language models like BERT excel at sentence classification tasks due to extensive pre-training on general data, but their robustness to parameter corruption is unexplored. To understand this better, we look at what happens if a language model is "broken", in the sense that some of its parameters are corrupted and then recovered by fine-tuning. Strategically corrupting BERT variants at different levels, we find corrupted models struggle to fully recover their original performance, with higher corruption causing more severe degradation. Notably, bottom-layer corruption affecting fundamental linguistic features is more detrimental than top-layer corruption. Our insights contribute to understanding language model robustness and adaptability under adverse conditions, informing strategies for developing resilient NLP systems against parameter perturbations. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2406_14459 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Healing Powers of BERT: How Task-Specific Fine-Tuning Recovers Corrupted Language Models Han, Shijie Zhang, Zhenyu Simion, Andrei Arsene Computation and Language Language models like BERT excel at sentence classification tasks due to extensive pre-training on general data, but their robustness to parameter corruption is unexplored. To understand this better, we look at what happens if a language model is "broken", in the sense that some of its parameters are corrupted and then recovered by fine-tuning. Strategically corrupting BERT variants at different levels, we find corrupted models struggle to fully recover their original performance, with higher corruption causing more severe degradation. Notably, bottom-layer corruption affecting fundamental linguistic features is more detrimental than top-layer corruption. Our insights contribute to understanding language model robustness and adaptability under adverse conditions, informing strategies for developing resilient NLP systems against parameter perturbations. |
| title | Healing Powers of BERT: How Task-Specific Fine-Tuning Recovers Corrupted Language Models |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2406.14459 |