Cover Image

MONA: Muon Optimizer with Nesterov Acceleration for Scalable Language Model Training

Fuente: arXiv

Saved in:

Bibliographic Details
Main Authors:	Li, Jiacheng, Tan, Jianchao, Xu, Hongtao, Zhang, Jiaqi, Lu, Yifan, Sun, Yerui, Xie, Yuchen, Cai, Xunliang
Format:	Preprint
Published:	2026
Subjects:	Machine Learning Computation and Language
Online Access:	Acceder al recurso
Tags:	Add Tag No Tags, Be the first to tag this record!

Similar Items

AFA-LoRA: Enabling Non-Linear Adaptations in LoRA with Activation Function Annealing
by: Li, Jiacheng, et al.
Published: (2025)

Optimizing Native Sparse Attention with Latent Attention and Local Global Alternating Strategies
by: Hu, Yuxuan, et al.
Published: (2025)

WISCA: A Lightweight Model Transition Method to Improve LLM Training via Weight Scaling
by: Li, Jiacheng, et al.
Published: (2025)

FG$^2$-GDN: Enhancing Long-Context Gated Delta Networks with Doubly Fine-Grained Control
by: Sun, Pingwei, et al.
Published: (2026)

Accelerate Speculative Decoding with Sparse Computation in Verification
by: Wang, Jikai, et al.
Published: (2025)

AsyncTLS: Efficient Generative LLM Inference with Asynchronous Two-level Sparse Attention
by: Hu, Yuxuan, et al.
Published: (2026)

SparseBalance: Load-Balanced Long Context Training with Dynamic Sparse Attention
by: Xu, Hongtao, et al.
Published: (2026)

Scaling Embeddings Outperforms Scaling Experts in Language Models
by: Liu, Hong, et al.
Published: (2026)

Muon is Scalable for LLM Training
by: Liu, Jingyuan, et al.
Published: (2025)

MaskPrune: Mask-based LLM Pruning for Layer-wise Uniform Structures
by: Qin, Jiayu, et al.
Published: (2025)

SnapMLA: Efficient Long-Context MLA Decoding via Hardware-Aware FP8 Quantized Pipelining
by: Zhang, Yifan, et al.
Published: (2026)

C2T: A Classifier-Based Tree Construction Method in Speculative Decoding
by: Huo, Feiye, et al.
Published: (2025)

MuonAll: Muon Variant for Efficient Finetuning of Large Language Models
by: Page, Saurabh, et al.
Published: (2025)

DORA: A Scalable Asynchronous Reinforcement Learning System for Language Model Training
by: Hu, Tianhao, et al.
Published: (2026)

When and Why Grouping Attention Heads Accelerates Muon Optimization
by: Zhang, Hongtao, et al.
Published: (2026)

EMA-Nesterov: Stabilizing Nesterov's Lookahead for Accelerated Deep Learning Optimization
by: Yau, Chung-Yiu, et al.
Published: (2026)

MiniCPM: Unveiling the Potential of Small Language Models with Scalable Training Strategies
by: Hu, Shengding, et al.
Published: (2024)

A Preliminary Study on the Promises and Challenges of Native Top-$k$ Sparse Attention
by: Xiu, Di, et al.
Published: (2025)

Leveraging Group Relative Policy Optimization to Advance Large Language Models in Traditional Chinese Medicine
by: Xie, Jiacheng, et al.
Published: (2025)

TopoCurate:Modeling Interaction Topology for Tool-Use Agent Training
by: Yang, Jinluan, et al.
Published: (2026)

Length Desensitization in Direct Preference Optimization
by: Liu, Wei, et al.
Published: (2024)

Ltri-LLM: Streaming Long Context Inference for LLMs with Training-Free Dynamic Triangular Attention Pattern
by: Tang, Hongyin, et al.
Published: (2024)

Training and Evaluating Language Models with Template-based Data Generation
by: Zhang, Yifan
Published: (2024)

Robust and Scalable Model Editing for Large Language Models
by: Chen, Yingfa, et al.
Published: (2024)

SpanNorm: Reconciling Training Stability and Performance in Deep Transformers
by: Wang, Chao, et al.
Published: (2026)

Adaptive Nesterov Accelerated Distributional Deep Hedging for Efficient Volatility Risk Management
by: Zhao, Lei, et al.
Published: (2025)

SVIP: Towards Verifiable Inference of Open-source Large Language Models
by: Sun, Yifan, et al.
Published: (2024)

Unraveling the Mystery of Scaling Laws: Part I
by: Su, Hui, et al.
Published: (2024)

NorMuon: Making Muon more efficient and scalable
by: Li, Zichong, et al.
Published: (2025)

Scalable Data Ablation Approximations for Language Models through Modular Training and Merging
by: Na, Clara, et al.
Published: (2024)

Understanding and Accelerating the Training of Masked Diffusion Language Models
by: Hong, Chunsan, et al.
Published: (2026)

BiTA: Bi-Directional Tuning for Lossless Acceleration in Large Language Models
by: Lin, Feng, et al.
Published: (2024)

Sparsity-Accelerated Training for Large Language Models
by: Ma, Da, et al.
Published: (2024)

Accelerated Preference Optimization for Large Language Model Alignment
by: He, Jiafan, et al.
Published: (2024)

Multi-Objective and Mixed-Reward Reinforcement Learning via Reward-Decorrelated Policy Optimization
by: Bai, Yang, et al.
Published: (2026)

Integer Scale: A Free Lunch for Faster Fine-grained Quantization of LLMs
by: Li, Qingyuan, et al.
Published: (2024)

Simple, Efficient and Scalable Structure-aware Adapter Boosts Protein Language Models
by: Tan, Yang, et al.
Published: (2024)

Scalable Efficient Training of Large Language Models with Low-dimensional Projected Attention
by: Lv, Xingtai, et al.
Published: (2024)

Dynamic Adaptation of LoRA Fine-Tuning for Efficient and Task-Specific Optimization of Large Language Models
by: Liao, Xiaoxuan, et al.
Published: (2025)

End-to-end Planner Training for Language Modeling
by: Cornille, Nathan, et al.
Published: (2024)