The power of fine-grained experts: Granularity boosts expressivity in Mixture of Experts
Fuente:
arXiv
Saved in:
| Main Authors: | , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866916731644018688 |
|---|---|
| author | Boix-Adsera, Enric Rigollet, Philippe |
| author_facet | Boix-Adsera, Enric Rigollet, Philippe |
| contents | Mixture-of-Experts (MoE) layers are increasingly central to frontier model architectures. By selectively activating parameters, they reduce computational cost while scaling total parameter count. This paper investigates the impact of the number of active experts, termed granularity, comparing architectures with many (e.g., 8 per layer in DeepSeek) to those with fewer (e.g., 1 per layer in Llama-4 models). We prove an exponential separation in network expressivity based on this design parameter, suggesting that models benefit from higher granularity. Experimental results corroborate our theoretical findings and illustrate this separation. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2505_06839 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | The power of fine-grained experts: Granularity boosts expressivity in Mixture of Experts Boix-Adsera, Enric Rigollet, Philippe Machine Learning Artificial Intelligence Mixture-of-Experts (MoE) layers are increasingly central to frontier model architectures. By selectively activating parameters, they reduce computational cost while scaling total parameter count. This paper investigates the impact of the number of active experts, termed granularity, comparing architectures with many (e.g., 8 per layer in DeepSeek) to those with fewer (e.g., 1 per layer in Llama-4 models). We prove an exponential separation in network expressivity based on this design parameter, suggesting that models benefit from higher granularity. Experimental results corroborate our theoretical findings and illustrate this separation. |
| title | The power of fine-grained experts: Granularity boosts expressivity in Mixture of Experts |
| topic | Machine Learning Artificial Intelligence |
| url | https://arxiv.org/abs/2505.06839 |