Noise-Robustness Through Noise: A Framework combining Asymmetric LoRA with Poisoning MoE
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2025
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866915564829540352 |
|---|---|
| author | Wang, Zhaokun Guo, Jinyu Pu, Jingwen Chen, Lingfeng Pu, Hongli Ou, Jie Qin, Libo Tian, Wenhong |
| author_facet | Wang, Zhaokun Guo, Jinyu Pu, Jingwen Chen, Lingfeng Pu, Hongli Ou, Jie Qin, Libo Tian, Wenhong |
| contents | Current parameter-efficient fine-tuning methods for adapting pre-trained language models to downstream tasks are susceptible to interference from noisy data. Conventional noise-handling approaches either rely on laborious data pre-processing or employ model architecture modifications prone to error accumulation. In contrast to existing noise-process paradigms, we propose a noise-robust adaptation method via asymmetric LoRA poisoning experts (LoPE), a novel framework that enhances model robustness to noise only with generated noisy data. Drawing inspiration from the mixture-of-experts architecture, LoPE strategically integrates a dedicated poisoning expert in an asymmetric LoRA configuration. Through a two-stage paradigm, LoPE performs noise injection on the poisoning expert during fine-tuning to enhance its noise discrimination and processing ability. During inference, we selectively mask the dedicated poisoning expert to leverage purified knowledge acquired by normal experts for noise-robust output. Extensive experiments demonstrate that LoPE achieves strong performance and robustness purely through the low-cost noise injection, which completely eliminates the requirement of data cleaning. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2505_23868 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Noise-Robustness Through Noise: A Framework combining Asymmetric LoRA with Poisoning MoE Wang, Zhaokun Guo, Jinyu Pu, Jingwen Chen, Lingfeng Pu, Hongli Ou, Jie Qin, Libo Tian, Wenhong Machine Learning Artificial Intelligence Current parameter-efficient fine-tuning methods for adapting pre-trained language models to downstream tasks are susceptible to interference from noisy data. Conventional noise-handling approaches either rely on laborious data pre-processing or employ model architecture modifications prone to error accumulation. In contrast to existing noise-process paradigms, we propose a noise-robust adaptation method via asymmetric LoRA poisoning experts (LoPE), a novel framework that enhances model robustness to noise only with generated noisy data. Drawing inspiration from the mixture-of-experts architecture, LoPE strategically integrates a dedicated poisoning expert in an asymmetric LoRA configuration. Through a two-stage paradigm, LoPE performs noise injection on the poisoning expert during fine-tuning to enhance its noise discrimination and processing ability. During inference, we selectively mask the dedicated poisoning expert to leverage purified knowledge acquired by normal experts for noise-robust output. Extensive experiments demonstrate that LoPE achieves strong performance and robustness purely through the low-cost noise injection, which completely eliminates the requirement of data cleaning. |
| title | Noise-Robustness Through Noise: A Framework combining Asymmetric LoRA with Poisoning MoE |
| topic | Machine Learning Artificial Intelligence |
| url | https://arxiv.org/abs/2505.23868 |