Steering MoE LLMs via Expert (De)Activation
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2025
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866918333151969280 |
|---|---|
| author | Fayyaz, Mohsen Modarressi, Ali Deilamsalehy, Hanieh Dernoncourt, Franck Rossi, Ryan Bui, Trung Schütze, Hinrich Peng, Nanyun |
| author_facet | Fayyaz, Mohsen Modarressi, Ali Deilamsalehy, Hanieh Dernoncourt, Franck Rossi, Ryan Bui, Trung Schütze, Hinrich Peng, Nanyun |
| contents | Mixture-of-Experts (MoE) in Large Language Models (LLMs) routes each token through a subset of specialized Feed-Forward Networks (FFN), known as experts. We present SteerMoE, a framework to steer MoE models by detecting and controlling behavior-associated experts. We detect key experts by comparing how often they activate between paired inputs that demonstrate opposite behaviors (e.g., safe vs. unsafe). By selectively activating or deactivating such experts during inference, we control behaviors like faithfulness and safety without fine-tuning. Across 11 benchmarks and 6 LLMs, our steering raises safety by up to +20% and faithfulness by +27%. Alternatively, unsafe steering drops safety by -41% alone, and -100% when combined with existing jailbreak methods, bypassing all safety guardrails. Overall, SteerMoE offers a lightweight, effective, and widely applicable test-time control, while revealing unique vulnerabilities in MoE LLMs. https://github.com/adobe-research/SteerMoE |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2509_09660 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Steering MoE LLMs via Expert (De)Activation Fayyaz, Mohsen Modarressi, Ali Deilamsalehy, Hanieh Dernoncourt, Franck Rossi, Ryan Bui, Trung Schütze, Hinrich Peng, Nanyun Computation and Language Machine Learning Mixture-of-Experts (MoE) in Large Language Models (LLMs) routes each token through a subset of specialized Feed-Forward Networks (FFN), known as experts. We present SteerMoE, a framework to steer MoE models by detecting and controlling behavior-associated experts. We detect key experts by comparing how often they activate between paired inputs that demonstrate opposite behaviors (e.g., safe vs. unsafe). By selectively activating or deactivating such experts during inference, we control behaviors like faithfulness and safety without fine-tuning. Across 11 benchmarks and 6 LLMs, our steering raises safety by up to +20% and faithfulness by +27%. Alternatively, unsafe steering drops safety by -41% alone, and -100% when combined with existing jailbreak methods, bypassing all safety guardrails. Overall, SteerMoE offers a lightweight, effective, and widely applicable test-time control, while revealing unique vulnerabilities in MoE LLMs. https://github.com/adobe-research/SteerMoE |
| title | Steering MoE LLMs via Expert (De)Activation |
| topic | Computation and Language Machine Learning |
| url | https://arxiv.org/abs/2509.09660 |