SUDA-Muon: Structural Design Principles and Boundaries for Fully Decentralized Muon
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2026
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866918469393448960 |
|---|---|
| author | Zhang, Hengrui Kong, Boao Geng, Jiahe Huang, Zhengyang |
| author_facet | Zhang, Hengrui Kong, Boao Geng, Jiahe Huang, Zhengyang |
| contents | Fully decentralized Muon is difficult because its nonlinear matrix-sign operator does not commute with linear gossip averaging. This makes decentralized Muon a structural design problem: in designing the algorithm, one must distinguish modular components from non-modular ones. We propose \sudamuon{}, which realizes this separation through a unified primal--dual communication template called SUDA; within this template, ED/D$^2$, EXTRA, and gradient tracking become modular backbone choices. We prove a topology-separated non-asymptotic convergence guarantee in the nuclear-norm geometry: the dominant term scales as $\mathcal{O}((1+σ/\sqrt{N})K^{-1/4})$ and does not explicitly involve graph quantities, identifying the communication backbone as the modular axis in the structure design. We then establish two complementary non-modular boundaries. Internally, tracking-before-polarization is necessary for this natural no-tracking variant to avoid non-stationary fixed points under heterogeneous objectives. Externally, in the absence of a central server, a fully decentralized method cannot perform the federated average-then-polarize update; we show that this non-modular local-polarize-then-average design is the essential reason why can fail to exhibit linear speedup. Experiments on CIFAR-100 and GPT-2 fine-tuning support the same picture: the unified template makes different communication algorithms directly comparable. In mild near-IID regimes, the resulting variants perform similarly, while in the more difficult long-horizon non-IID CIFAR-100 setting, \sudamuon{} achieves higher accuracy and lower loss than \textsc{DeMuon}. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2604_23980 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | SUDA-Muon: Structural Design Principles and Boundaries for Fully Decentralized Muon Zhang, Hengrui Kong, Boao Geng, Jiahe Huang, Zhengyang Optimization and Control 90C15, 90C26, 90C35, 68W15 Fully decentralized Muon is difficult because its nonlinear matrix-sign operator does not commute with linear gossip averaging. This makes decentralized Muon a structural design problem: in designing the algorithm, one must distinguish modular components from non-modular ones. We propose \sudamuon{}, which realizes this separation through a unified primal--dual communication template called SUDA; within this template, ED/D$^2$, EXTRA, and gradient tracking become modular backbone choices. We prove a topology-separated non-asymptotic convergence guarantee in the nuclear-norm geometry: the dominant term scales as $\mathcal{O}((1+σ/\sqrt{N})K^{-1/4})$ and does not explicitly involve graph quantities, identifying the communication backbone as the modular axis in the structure design. We then establish two complementary non-modular boundaries. Internally, tracking-before-polarization is necessary for this natural no-tracking variant to avoid non-stationary fixed points under heterogeneous objectives. Externally, in the absence of a central server, a fully decentralized method cannot perform the federated average-then-polarize update; we show that this non-modular local-polarize-then-average design is the essential reason why can fail to exhibit linear speedup. Experiments on CIFAR-100 and GPT-2 fine-tuning support the same picture: the unified template makes different communication algorithms directly comparable. In mild near-IID regimes, the resulting variants perform similarly, while in the more difficult long-horizon non-IID CIFAR-100 setting, \sudamuon{} achieves higher accuracy and lower loss than \textsc{DeMuon}. |
| title | SUDA-Muon: Structural Design Principles and Boundaries for Fully Decentralized Muon |
| topic | Optimization and Control 90C15, 90C26, 90C35, 68W15 |
| url | https://arxiv.org/abs/2604.23980 |