Multi-Head Attention as a Source of Catastrophic Forgetting in MoE Transformers
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2026
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866918335604588544 |
|---|---|
| author | Chen, Anrui Huang, Ruijun Zhang, Xin Dong, Fang Cao, Hengjie Huang, Zhendong Yang, Yifeng Chen, Mengyi Zhou, Jixian Dong, Mingzhi Wang, Yujiang Hou, Jinlong Lv, Qin Dick, Robert P. Cheng, Yuan Lu, Tun Yang, Fan Shang, Li |
| author_facet | Chen, Anrui Huang, Ruijun Zhang, Xin Dong, Fang Cao, Hengjie Huang, Zhendong Yang, Yifeng Chen, Mengyi Zhou, Jixian Dong, Mingzhi Wang, Yujiang Hou, Jinlong Lv, Qin Dick, Robert P. Cheng, Yuan Lu, Tun Yang, Fan Shang, Li |
| contents | Mixture-of-Experts (MoE) architectures are often considered a natural fit for continual learning because sparse routing should localize updates and reduce interference, yet MoE Transformers still forget substantially even with sparse, well-balanced expert utilization. We attribute this gap to a pre-routing bottleneck: multi-head attention concatenates head-specific signals into a single post-attention router input, forcing routing to act on co-occurring feature compositions rather than separable head channels. We show that this router input simultaneously encodes multiple separately decodable semantic and structural factors with uneven head support, and that different feature compositions induce weakly aligned parameter-gradient directions; as a result, routing maps many distinct compositions to the same route. We quantify this collision effect via a route-wise effective composition number $N_{eff}$ and find that higher $N_{eff}$ is associated with larger old-task loss increases after continual training. Motivated by these findings, we propose MH-MoE, which performs head-wise routing over sub-representations to increase routing granularity and reduce composition collisions. On TRACE with Qwen3-0.6B/8B, MH-MoE effectively mitigates forgetting, reducing BWT on Qwen3-0.6B from 11.2% (LoRAMoE) to 4.5%. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2602_12587 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | Multi-Head Attention as a Source of Catastrophic Forgetting in MoE Transformers Chen, Anrui Huang, Ruijun Zhang, Xin Dong, Fang Cao, Hengjie Huang, Zhendong Yang, Yifeng Chen, Mengyi Zhou, Jixian Dong, Mingzhi Wang, Yujiang Hou, Jinlong Lv, Qin Dick, Robert P. Cheng, Yuan Lu, Tun Yang, Fan Shang, Li Machine Learning Mixture-of-Experts (MoE) architectures are often considered a natural fit for continual learning because sparse routing should localize updates and reduce interference, yet MoE Transformers still forget substantially even with sparse, well-balanced expert utilization. We attribute this gap to a pre-routing bottleneck: multi-head attention concatenates head-specific signals into a single post-attention router input, forcing routing to act on co-occurring feature compositions rather than separable head channels. We show that this router input simultaneously encodes multiple separately decodable semantic and structural factors with uneven head support, and that different feature compositions induce weakly aligned parameter-gradient directions; as a result, routing maps many distinct compositions to the same route. We quantify this collision effect via a route-wise effective composition number $N_{eff}$ and find that higher $N_{eff}$ is associated with larger old-task loss increases after continual training. Motivated by these findings, we propose MH-MoE, which performs head-wise routing over sub-representations to increase routing granularity and reduce composition collisions. On TRACE with Qwen3-0.6B/8B, MH-MoE effectively mitigates forgetting, reducing BWT on Qwen3-0.6B from 11.2% (LoRAMoE) to 4.5%. |
| title | Multi-Head Attention as a Source of Catastrophic Forgetting in MoE Transformers |
| topic | Machine Learning |
| url | https://arxiv.org/abs/2602.12587 |