Image de couverture de livre

Mousse: Rectifying the Geometry of Muon with Curvature-Aware Preconditioning

Fuente: arXiv

Enregistré dans:

Détails bibliographiques
Auteurs principaux:	Zhang, Yechen, Xing, Shuhao, Huang, Junhao, Lv, Kai, Zhou, Yunhua, Qiu, Xipeng, Guo, Qipeng, Chen, Kai
Format:	Preprint
Publié:	2026
Sujets:	Machine Learning Artificial Intelligence Computation and Language
Accès en ligne:	Acceder al recurso
Tags:	Ajouter un tag Pas de tags, Soyez le premier à ajouter un tag!

Documents similaires

How to Set the Learning Rate for Large-Scale Pre-training?
par: Zhou, Yunhua, et autres
Publié: (2026)

Explicit Multi-head Attention for Inter-head Interaction in Large Language Models
par: Peng, Runyu, et autres
Publié: (2026)

Turn Waste into Worth: Rectifying Top-$k$ Router of MoE
par: Zeng, Zhiyuan, et autres
Publié: (2024)

How to Set the Batch Size for Large-Scale Pre-training?
par: Zhou, Yunhua, et autres
Publié: (2026)

Revisiting the Test-Time Scaling of o1-like Models: Do they Truly Possess Test-Time Scaling Capabilities?
par: Zeng, Zhiyuan, et autres
Publié: (2025)

Implicit Reward as the Bridge: A Unified View of SFT and DPO Connections
par: Wang, Bo, et autres
Publié: (2025)

Data Mixing Laws: Optimizing Data Mixtures by Predicting Language Modeling Performance
par: Ye, Jiasheng, et autres
Publié: (2024)

AdaLomo: Low-memory Optimization with Adaptive Learning Rate
par: Lv, Kai, et autres
Publié: (2023)

How to Mitigate Overfitting in Weak-to-strong Generalization?
par: Shi, Junhao, et autres
Publié: (2025)

Scaling of Search and Learning: A Roadmap to Reproduce o1 from Reinforcement Learning Perspective
par: Zeng, Zhiyuan, et autres
Publié: (2024)

How Attention Sinks Emerge in Large Language Models: An Interpretability Perspective
par: Peng, Runyu, et autres
Publié: (2026)

BitStack: Any-Size Compression of Large Language Models in Variable Memory Environments
par: Wang, Xinghao, et autres
Publié: (2024)

Preconditioning Benefits of Spectral Orthogonalization in Muon
par: Ma, Jianhao, et autres
Publié: (2026)

ReAttention: Training-Free Infinite Context with Finite Attention Scope
par: Liu, Xiaoran, et autres
Publié: (2024)

DuoDecoding: Hardware-aware Heterogeneous Speculative Decoding with Dynamic Multi-Sequence Drafting
par: Lv, Kai, et autres
Publié: (2025)

LongSafety: Enhance Safety for Long-Context LLMs
par: Huang, Mianqiu, et autres
Publié: (2024)

Evolution of Concepts in Language Model Pre-Training
par: Ge, Xuyang, et autres
Publié: (2025)

Muon$^2$: Boosting Muon via Adaptive Second-Moment Preconditioning
par: Liu, Ziyue, et autres
Publié: (2026)

Rectifying LLM Thought from Lens of Optimization
par: Liu, Junnan, et autres
Publié: (2025)

Identifying Semantic Induction Heads to Understand In-Context Learning
par: Ren, Jie, et autres
Publié: (2024)

E-Sparse: Boosting the Large Language Model Inference through Entropy-based N:M Sparsity
par: Li, Yun, et autres
Publié: (2023)

Domain2Vec: Vectorizing Datasets to Find the Optimal Data Mixture without Training
par: Zhang, Mozhi, et autres
Publié: (2025)

Towards Economical Inference: Enabling DeepSeek's Multi-Head Latent Attention in Any Transformer-based LLMs
par: Ji, Tao, et autres
Publié: (2025)

Timely Machine: Awareness of Time Makes Test-Time Scaling Agentic
par: Ma, Yichuan, et autres
Publié: (2026)

Muon is Scalable for LLM Training
par: Liu, Jingyuan, et autres
Publié: (2025)

MetaMoE: Diversity-Aware Proxy Selection for Privacy-Preserving Mixture-of-Experts Unification
par: Jiang, Weisen, et autres
Publié: (2026)

Context-Aware Initialization for Reducing Generative Path Length in Diffusion Language Models
par: Miao, Tongyuan, et autres
Publié: (2025)

Delving into Muon and Beyond: Deep Analysis and Extensions
par: Qi, Xianbiao, et autres
Publié: (2026)

Selecting Large Language Model to Fine-tune via Rectified Scaling Law
par: Lin, Haowei, et autres
Publié: (2024)

Memory Decoder: A Pretrained, Plug-and-Play Memory for Large Language Models
par: Cao, Jiaqi, et autres
Publié: (2025)

World-aware Planning Narratives Enhance Large Vision-Language Model Planner
par: Shi, Junhao, et autres
Publié: (2025)

Unleashing Embodied Task Planning Ability in LLMs via Reinforcement Learning
par: Fei, Zhaoye, et autres
Publié: (2025)

Pre-Trained Policy Discriminators are General Reward Models
par: Dou, Shihan, et autres
Publié: (2025)

Prism: Spectral-Aware Block-Sparse Attention
par: Wang, Xinghao, et autres
Publié: (2026)

MoST: Mixing Speech and Text with Modality-Aware Mixture of Experts
par: Lou, Yuxuan, et autres
Publié: (2026)

What are the Essential Factors in Crafting Effective Long Context Multi-Hop Instruction Datasets? Insights and Best Practices
par: Chen, Zhi, et autres
Publié: (2024)

CARL: Criticality-Aware Agentic Reinforcement Learning
par: Shen, Leyang, et autres
Publié: (2025)

AdaPreLoRA: Adafactor Preconditioned Low-Rank Adaptation
par: Liu, Ziyun, et autres
Publié: (2026)

Integrating Locality-Aware Attention with Transformers for General Geometry PDEs
par: Koh, Minsu, et autres
Publié: (2025)

UltraEdit: Training-, Subject-, and Memory-Free Lifelong Editing in Language Models
par: Gu, Xiaojie, et autres
Publié: (2025)