Imagen de Portada

How to Set the Learning Rate for Large-Scale Pre-training?

Fuente: arXiv

Guardado en:

Detalles Bibliográficos
Autores principales:	Zhou, Yunhua, Xing, Shuhao, Huang, Junhao, Qiu, Xipeng, Guo, Qipeng
Formato:	Preprint
Publicado:	2026
Materias:	Artificial Intelligence
Acceso en línea:	Acceder al recurso
Etiquetas:	Agregar Etiqueta Sin Etiquetas, Sea el primero en etiquetar este registro!

Ejemplares similares

How to Set the Batch Size for Large-Scale Pre-training?
por: Zhou, Yunhua, et al.
Publicado: (2026)

Mousse: Rectifying the Geometry of Muon with Curvature-Aware Preconditioning
por: Zhang, Yechen, et al.
Publicado: (2026)

Scaling of Search and Learning: A Roadmap to Reproduce o1 from Reinforcement Learning Perspective
por: Zeng, Zhiyuan, et al.
Publicado: (2024)

How to Mitigate Overfitting in Weak-to-strong Generalization?
por: Shi, Junhao, et al.
Publicado: (2025)

Explicit Multi-head Attention for Inter-head Interaction in Large Language Models
por: Peng, Runyu, et al.
Publicado: (2026)

World-aware Planning Narratives Enhance Large Vision-Language Model Planner
por: Shi, Junhao, et al.
Publicado: (2025)

ARISE: An Adaptive Resolution-Aware Metric for Test-Time Scaling Evaluation in Large Reasoning Models
por: Yin, Zhangyue, et al.
Publicado: (2025)

Evolution of Concepts in Language Model Pre-Training
por: Ge, Xuyang, et al.
Publicado: (2025)

Revisiting the Test-Time Scaling of o1-like Models: Do they Truly Possess Test-Time Scaling Capabilities?
por: Zeng, Zhiyuan, et al.
Publicado: (2025)

Turn Waste into Worth: Rectifying Top-$k$ Router of MoE
por: Zeng, Zhiyuan, et al.
Publicado: (2024)

Implicit Reward as the Bridge: A Unified View of SFT and DPO Connections
por: Wang, Bo, et al.
Publicado: (2025)

Data Darwinism Part I: Unlocking the Value of Scientific Data for Pre-training
por: Qin, Yiwei, et al.
Publicado: (2026)

Identifying Semantic Induction Heads to Understand In-Context Learning
por: Ren, Jie, et al.
Publicado: (2024)

How Attention Sinks Emerge in Large Language Models: An Interpretability Perspective
por: Peng, Runyu, et al.
Publicado: (2026)

BitStack: Any-Size Compression of Large Language Models in Variable Memory Environments
por: Wang, Xinghao, et al.
Publicado: (2024)

Data Mixing Laws: Optimizing Data Mixtures by Predicting Language Modeling Performance
por: Ye, Jiasheng, et al.
Publicado: (2024)

How to inject knowledge efficiently? Knowledge Infusion Scaling Law for Pre-training Large Language Models
por: Lv, Kangtao, et al.
Publicado: (2025)

Unleashing Embodied Task Planning Ability in LLMs via Reinforcement Learning
por: Fei, Zhaoye, et al.
Publicado: (2025)

Emergent Structured Representations Support Flexible In-Context Inference in Large Language Models
por: Xu, Ningyu, et al.
Publicado: (2026)

How Does Critical Batch Size Scale in Pre-training?
por: Zhang, Hanlin, et al.
Publicado: (2024)

Towards Economical Inference: Enabling DeepSeek's Multi-Head Latent Attention in Any Transformer-based LLMs
por: Ji, Tao, et al.
Publicado: (2025)

Backdoor in Seconds: Unlocking Vulnerabilities in Large Pre-trained Models via Model Editing
por: Guo, Dongliang, et al.
Publicado: (2024)

MSR-HuBERT: Self-supervised Pre-training for Adaptation to Multiple Sampling Rates
por: Huang, Zikang, et al.
Publicado: (2026)

Aligning Instruction Tuning with Pre-training
por: Liang, Yiming, et al.
Publicado: (2025)

Low-hallucination Synthetic Captions for Large-Scale Vision-Language Model Pre-training
por: Zhang, Xinsong, et al.
Publicado: (2025)

RefineX: Learning to Refine Pre-training Data at Scale from Expert-Guided Programs
por: Bi, Baolong, et al.
Publicado: (2025)

Scaling LLM Pre-training with Vocabulary Curriculum
por: Yu, Fangyuan
Publicado: (2025)

RLoop: An Self-Improving Framework for Reinforcement Learning with Iterative Policy Initialization
por: Zhiyuan, Zeng, et al.
Publicado: (2025)

Learning Top-k Subtask Planning Tree based on Discriminative Representation Pre-training for Decision Making
por: Ruan, Jingqing, et al.
Publicado: (2023)

Scaling Laws of RoPE-based Extrapolation
por: Liu, Xiaoran, et al.
Publicado: (2023)

OSF: On Pre-training and Scaling of Sleep Foundation Models
por: Shuai, Zitao, et al.
Publicado: (2026)

Scaling Laws for Pre-training Agents and World Models
por: Pearce, Tim, et al.
Publicado: (2024)

ReAttention: Training-Free Infinite Context with Finite Attention Scope
por: Liu, Xiaoran, et al.
Publicado: (2024)

GraphPro: Graph Pre-training and Prompt Learning for Recommendation
por: Yang, Yuhao, et al.
Publicado: (2023)

Pre-trained Large Language Models Learn Hidden Markov Models In-context
por: Dai, Yijia, et al.
Publicado: (2025)

Leveraging Pre-trained Large Language Models with Refined Prompting for Online Task and Motion Planning
por: Guo, Huihui, et al.
Publicado: (2025)

Robots Pre-train Robots: Manipulation-Centric Robotic Representation from Large-Scale Robot Datasets
por: Jiang, Guangqi, et al.
Publicado: (2024)

PPM : A Pre-trained Plug-in Model for Click-through Rate Prediction
por: Gao, Yuanbo, et al.
Publicado: (2024)

Advancing ALS Applications with Large-Scale Pre-training: Dataset Development and Downstream Assessment
por: Xiu, Haoyi, et al.
Publicado: (2025)

Scaling Smart: Accelerating Large Language Model Pre-training with Small Model Initialization
por: Samragh, Mohammad, et al.
Publicado: (2024)