Imagen de Portada

Taming LLMs by Scaling Learning Rates with Gradient Grouping

Fuente: arXiv

Guardado en:

Detalles Bibliográficos
Autores principales:	Li, Siyuan, Tian, Juanxi, Wang, Zedong, Jin, Xin, Liu, Zicheng, Zhang, Wentao, Xu, Dan
Formato:	Preprint
Publicado:	2025
Materias:	Machine Learning Artificial Intelligence
Acceso en línea:	Acceder al recurso
Etiquetas:	Agregar Etiqueta Sin Etiquetas, Sea el primero en etiquetar este registro!

Ejemplares similares

A Survey on Mixup Augmentations and Beyond
por: Jin, Xin, et al.
Publicado: (2024)

SemiReward: A General Reward Model for Semi-supervised Learning
por: Li, Siyuan, et al.
Publicado: (2023)

EMA Policy Gradient: Taming Reinforcement Learning for LLMs with EMA Anchor and Top-k KL
por: Zhang, Lunjun, et al.
Publicado: (2026)

Unveiling the Backbone-Optimizer Coupling Bias in Visual Representation Learning
por: Li, Siyuan, et al.
Publicado: (2024)

APRIL: Active Partial Rollouts in Reinforcement Learning to Tame Long-tail Generation
por: Zhou, Yuzhen, et al.
Publicado: (2025)

Dynamic Vocabulary Pruning: Stable LLM-RL by Taming the Tail
por: Li, Yingru, et al.
Publicado: (2025)

EFRame: Deeper Reasoning via Exploration-Filter-Replay Reinforcement Learning Framework
por: Wang, Chen, et al.
Publicado: (2025)

Continual Gradient Low-Rank Projection Fine-Tuning for LLMs
por: Wang, Chenxu, et al.
Publicado: (2025)

Reasoning as Gradient: Scaling MLE Agents Beyond Tree Search
por: Zhang, Yifei, et al.
Publicado: (2026)

Taming Preconditioner Drift: Unlocking the Potential of Second-Order Optimizers for Federated Learning on Non-IID Data
por: Liu, Junkang, et al.
Publicado: (2026)

CurES: From Gradient Analysis to Efficient Curriculum Learning for Reasoning LLMs
por: Zeng, Yongcheng, et al.
Publicado: (2025)

RDesign: Hierarchical Data-efficient Representation Learning for Tertiary Structure-based RNA Design
por: Tan, Cheng, et al.
Publicado: (2023)

No More Adam: Learning Rate Scaling at Initialization is All You Need
por: Xu, Minghao, et al.
Publicado: (2024)

Kimi k1.5: Scaling Reinforcement Learning with LLMs
por: Kimi Team, et al.
Publicado: (2025)

GAC: Stabilizing Asynchronous RL Training for LLMs via Gradient Alignment Control
por: Xu, Haofeng, et al.
Publicado: (2026)

Taming the Adversary: Stable Minimax Deep Deterministic Policy Gradient via Fractional Objectives
por: Lee, Taeho, et al.
Publicado: (2026)

Physics of Language Models: Part 2.2, How to Learn From Mistakes on Grade-School Math Problems
por: Ye, Tian, et al.
Publicado: (2024)

GradientStabilizer:Fix the Norm, Not the Gradient
por: Huang, Tianjin, et al.
Publicado: (2025)

CriticBench: Benchmarking LLMs for Critique-Correct Reasoning
por: Lin, Zicheng, et al.
Publicado: (2024)

The Reciprocity Gradient
por: Lin, Yue, et al.
Publicado: (2026)

Conflict-Averse Gradient Descent for Multi-task Learning
por: Liu, Bo, et al.
Publicado: (2021)

One LR Doesn't Fit All: Heavy-Tail Guided Layerwise Learning Rates for LLMs
por: He, Di, et al.
Publicado: (2026)

Towards Mitigation of Hallucination for LLM-empowered Agents: Progressive Generalization Bound Exploration and Watchdog Monitor
por: Liu, Siyuan, et al.
Publicado: (2025)

Scaling Law with Learning Rate Annealing
por: Tissue, Howe, et al.
Publicado: (2024)

GaLore 2: Large-Scale LLM Pre-Training by Gradient Low-Rank Projection
por: Su, DiJia, et al.
Publicado: (2025)

Design Conditions for Intra-Group Learning of Sequence-Level Rewards: Token Gradient Cancellation
por: Ding, Fei, et al.
Publicado: (2026)

PEARL: Unbiased Percentile Estimation via Contrastive Learning for Industrial-Scale Livestream Recommendation
por: Gella, Blake, et al.
Publicado: (2026)

Enhancing DP-SGD through Non-monotonous Adaptive Scaling Gradient Weight
por: Huang, Tao, et al.
Publicado: (2024)

Model Failure or Data Corruption? Exploring Inconsistencies in Building Energy Ratings with Self-Supervised Contrastive Learning
por: Xiao, Qian, et al.
Publicado: (2024)

Metric-Gradient Projection for Stable Multi-Agent Policy Learning
por: Zhang, Zuyuan, et al.
Publicado: (2026)

True Knowledge Comes from Practice: Aligning LLMs with Embodied Environments via Reinforcement Learning
por: Tan, Weihao, et al.
Publicado: (2024)

Revealing Modular Gradient Noise Imbalance in LLMs: Calibrating Adam via Signal-to-Noise Ratio
por: Wen, Ziqing, et al.
Publicado: (2026)

Memory-Efficient Gradient Unrolling for Large-Scale Bi-level Optimization
por: Shen, Qianli, et al.
Publicado: (2024)

Rep-MTL: Unleashing the Power of Representation-level Task Saliency for Multi-Task Learning
por: Wang, Zedong, et al.
Publicado: (2025)

Diffusion LLMs Can Do Faster-Than-AR Inference via Discrete Diffusion Forcing
por: Wang, Xu, et al.
Publicado: (2025)

Optimize Weight Rounding via Signed Gradient Descent for the Quantization of LLMs
por: Cheng, Wenhua, et al.
Publicado: (2023)

Gradient-Congruity Guided Federated Sparse Training
por: Tian, Chris Xing, et al.
Publicado: (2024)

Optimal Embedding Learning Rate in LLMs: The Effect of Vocabulary Size
por: Hayou, Soufiane, et al.
Publicado: (2025)

Taming the Real-world Complexities in CPT E/M Coding with Large Language Models
por: Nassar, Islam, et al.
Publicado: (2025)

From $\log π$ to $π$: Taming Divergence in Soft Clipping via Bilateral Decoupled Decay of Probability Gradient Weight
por: Fu, Xiaoliang, et al.
Publicado: (2026)