Copertina

Understanding Quantization of Optimizer States in LLM Pre-training: Dynamics of State Staleness and Effectiveness of State Resets

Fuente: arXiv

Salvato in:

Dettagli Bibliografici
Autori principali:	Topollai, Kristi, Choromanska, Anna
Natura:	Preprint
Pubblicazione:	2026
Soggetti:	Machine Learning
Accesso online:	Acceder al recurso
Tags:	Aggiungi Tag Nessun Tag, puoi essere il primo ad aggiungerne!!

Documenti analoghi

Adaptive Memory Momentum via a Model-Based Framework for Deep Learning Optimization
di: Topollai, Kristi, et al.
Pubblicazione: (2025)

Task-Level Contrastiveness for Cross-Domain Few-Shot Learning
di: Topollai, Kristi, et al.
Pubblicazione: (2025)

Worker Disagreement Reveals Sharp Directions in Local SGD
di: Dimlioglu, Tolga, et al.
Pubblicazione: (2026)

Outer-Momentum Restarting in High-Dimensional Two-Phase Optimization
di: Topollai, Kristi, et al.
Pubblicazione: (2026)

OncoReason: Structuring Clinical Reasoning in LLMs for Robust and Interpretable Survival Prediction
di: Hemadri, Raghu Vamshi, et al.
Pubblicazione: (2025)

Effective Quantization of Muon Optimizer States
di: Gupta, Aman, et al.
Pubblicazione: (2025)

Streamlining Industrial Contract Management with Retrieval-Augmented LLMs
di: Topollai, Kristi, et al.
Pubblicazione: (2025)

Self-Supervised Representation Learning with Joint Embedding Predictive Architecture for Automotive LiDAR Object Detection
di: Zhu, Haoran, et al.
Pubblicazione: (2025)

A Survey of Optimization Methods for Training DL Models: Theoretical Perspective on Convergence and Generalization
di: Wang, Jing, et al.
Pubblicazione: (2025)

Communication-Efficient Distributed Training for Collaborative Flat Optima Recovery in Deep Learning
di: Dimlioglu, Tolga, et al.
Pubblicazione: (2025)

This Too Shall Pass: Removing Stale Observations in Dynamic Bayesian Optimization
di: Bardou, Anthony, et al.
Pubblicazione: (2024)

Adjacent Leader Decentralized Stochastic Gradient Descent
di: He, Haoze, et al.
Pubblicazione: (2024)

The Fourth State: Signed-Zero Ternary for Stable LLM Quantization (and More)
di: Uhlmann, Jeffrey
Pubblicazione: (2025)

GRAWA: Gradient-based Weighted Averaging for Distributed Training of Deep Learning Models
di: Dimlioglu, Tolga, et al.
Pubblicazione: (2024)

Priming: Hybrid State Space Models From Pre-trained Transformers
di: Chattopadhyay, Aditya, et al.
Pubblicazione: (2026)

Sample-efficient LLM Optimization with Reset Replay
di: Liu, Zichuan, et al.
Pubblicazione: (2025)

TAME: Task Agnostic Continual Learning using Multiple Experts
di: Zhu, Haoran, et al.
Pubblicazione: (2022)

Accelerating Recommender Model Training by Dynamically Skipping Stale Embeddings
di: Maboud, Yassaman Ebrahimzadeh, et al.
Pubblicazione: (2024)

Quantifying Memory Utilization with Effective State-Size
di: Parnichkun, Rom N., et al.
Pubblicazione: (2025)

State Regularized Policy Optimization on Data with Dynamics Shift
di: Xue, Zhenghai, et al.
Pubblicazione: (2023)

MUON+: Towards More Effective Muon via One Additional Normalization Step for LLM Pre-training
di: Zhang, Ruijie, et al.
Pubblicazione: (2026)

Discrete Semantic States and Hamiltonian Dynamics in LLM Embedding Spaces
di: Laine, Timo Aukusti
Pubblicazione: (2025)

Quamba2: A Robust and Scalable Post-training Quantization Framework for Selective State Space Models
di: Chiang, Hung-Yueh, et al.
Pubblicazione: (2025)

StateX: Enhancing RNN Recall via Post-training State Expansion
di: Shen, Xingyu, et al.
Pubblicazione: (2025)

Zero-Shot Cross-City Generalization in End-to-End Autonomous Driving: Self-Supervised versus Supervised Representations
di: Naeinian, Fatemeh, et al.
Pubblicazione: (2026)

Quantizing Small-Scale State-Space Models for Edge AI
di: Zhao, Leo, et al.
Pubblicazione: (2025)

Pre-trained Gaussian Processes for Bayesian Optimization
di: Wang, Zi, et al.
Pubblicazione: (2021)

Latent State Models of Training Dynamics
di: Hu, Michael Y., et al.
Pubblicazione: (2023)

FedStaleWeight: Buffered Asynchronous Federated Learning with Fair Aggregation via Staleness Reweighting
di: Ma, Jeffrey, et al.
Pubblicazione: (2024)

Absolute State-wise Constrained Policy Optimization: High-Probability State-wise Constraints Satisfaction
di: Zhao, Weiye, et al.
Pubblicazione: (2024)

Cosine-Gated Adam-Decay: Drop-In Staleness-Aware Outer Optimization for Decoupled DiLoCo
di: Shah, Vatsal, et al.
Pubblicazione: (2026)

State-wise Constrained Policy Optimization
di: Zhao, Weiye, et al.
Pubblicazione: (2023)

Tahakom LLM Guidelines and Recipes: From Pre-training Data to an Arabic LLM
di: AlOtaibi, Areej, et al.
Pubblicazione: (2025)

Communication Efficient LLM Pre-training with SparseLoCo
di: Sarfi, Amir, et al.
Pubblicazione: (2025)

Q-S5: Towards Quantized State Space Models
di: Abreu, Steven, et al.
Pubblicazione: (2024)

GeoDynamics: A Geometric State-Space Neural Network for Understanding Brain Dynamics on Riemannian Manifolds
di: Dan, Tingting, et al.
Pubblicazione: (2026)

Markovian Circuit Tracing for Transformer State Dynamic
di: X, Abdullah
Pubblicazione: (2026)

Dynamical Survival Analysis with Controlled Latent States
di: Bleistein, Linus, et al.
Pubblicazione: (2024)

Dataset Reset Policy Optimization for RLHF
di: Chang, Jonathan D., et al.
Pubblicazione: (2024)

Hista and Numca: Estimate State Value Effectively for LLM Reinforcement Learning
di: Chen, Zizhe, et al.
Pubblicazione: (2026)