Cover Image

Checkpoint Merging via Bayesian Optimization in LLM Pretraining

Fuente: arXiv

Saved in:

Bibliographic Details
Main Authors:	Liu, Deyuan, Wang, Zecheng, Wang, Bingning, Chen, Weipeng, Li, Chunshan, Tu, Zhiying, Chu, Dianhui, Li, Bo, Sui, Dianbo
Format:	Preprint
Published:	2024
Subjects:	Computation and Language
Online Access:	Acceder al recurso
Tags:	Add Tag No Tags, Be the first to tag this record!

Similar Items

Gradients Must Earn Their Influence: Unifying SFT with Generalized Entropic Objectives
by: Wang, Zecheng, et al.
Published: (2026)

Beyond Confidence: The Rhythms of Reasoning in Generative Models
by: Liu, Deyuan, et al.
Published: (2026)

Pruning via Merging: Compressing LLMs via Manifold Alignment Based Layer Merging
by: Liu, Deyuan, et al.
Published: (2024)

Mitigating Gender Bias in Code Large Language Models via Model Editing
by: Qin, Zhanyue, et al.
Published: (2024)

LFTF: Locating First and Then Fine-Tuning for Mitigating Gender Bias in Large Language Models
by: Qin, Zhanyue, et al.
Published: (2025)

UNO Arena for Evaluating Sequential Decision-Making Capability of Large Language Models
by: Qin, Zhanyue, et al.
Published: (2024)

Surrogate Signals from Format and Length: Reinforcement Learning for Solving Mathematical Problems without Ground Truth Answers
by: Xin, Rihui, et al.
Published: (2025)

MetaGPT: Merging Large Language Models Using Model Exclusive Task Arithmetic
by: Zhou, Yuyan, et al.
Published: (2024)

A Survey on Data Selection for LLM Instruction Tuning
by: Zhang, Bolin, et al.
Published: (2024)

ScEdit: Script-based Assessment of Knowledge Editing
by: Li, Xinye, et al.
Published: (2025)

EchoReview: Learning Peer Review from the Echoes of Scientific Citations
by: Zhang, Yinuo, et al.
Published: (2026)

LLMSR@XLLM25: An Empirical Study of LLM for Structural Reasoning
by: Li, Xinye, et al.
Published: (2025)

KV Shifting Attention Enhances Language Modeling
by: Xu, Mingyu, et al.
Published: (2024)

HBot: A Chatbot for Healthcare Applications in Traditional Chinese Medicine Based on Human Body 3D Visualization
by: Zhang, Bolin, et al.
Published: (2024)

Beyond Filtering: Adaptive Image-Text Quality Enhancement for MLLM Pretraining
by: Huang, Han, et al.
Published: (2024)

RSRWKV: A Linear-Complexity 2D Attention Mechanism for Efficient Remote Sensing Vision Task
by: Li, Chunshan, et al.
Published: (2025)

UniPrefill: Universal Long-Context Prefill Acceleration via Block-wise Dynamic Sparsification
by: Fan, Qihang, et al.
Published: (2026)

Full-ECE: A Metric For Token-level Calibration on Large Language Models
by: Liu, Han, et al.
Published: (2024)

Dynamic Fisher-weighted Model Merging via Bayesian Optimization
by: Lee, Sanwoo, et al.
Published: (2025)

WSM: Decay-Free Learning Rate Schedule via Checkpoint Merging for LLM Pre-training
by: Tian, Changxin, et al.
Published: (2025)

LongReD: Mitigating Short-Text Degradation of Long-Context Large Language Models via Restoration Distillation
by: Dong, Zican, et al.
Published: (2025)

Tree-OPO: Off-policy Monte Carlo Tree-Guided Advantage Optimization for Multistep Reasoning
by: Huang, Bingning, et al.
Published: (2025)

Unveiling the Flaws: Exploring Imperfections in Synthetic Data and Mitigation Strategies for Large Language Models
by: Chen, Jie, et al.
Published: (2024)

FlashPrefill: Instantaneous Pattern Discovery and Thresholding for Ultra-Fast Long-Context Prefilling
by: Fan, Qihang, et al.
Published: (2026)

Base of RoPE Bounds Context Length
by: Men, Xin, et al.
Published: (2024)

A Framework for Effective Invocation Methods of Various LLM Services
by: Wang, Can, et al.
Published: (2024)

Extracting and Combining Abilities For Building Multi-lingual Ability-enhanced Large Language Models
by: Chen, Zhipeng, et al.
Published: (2024)

Divide, Optimize, Merge: Fine-Grained LLM Agent Optimization at Scale
by: Liu, Jiale, et al.
Published: (2025)

ShortGPT: Layers in Large Language Models are More Redundant Than You Expect
by: Men, Xin, et al.
Published: (2024)

PlaM: Training-Free Plateau-Guided Model Merging for Better Visual Grounding in MLLMs
by: Wang, Zijing, et al.
Published: (2026)

Dissecting Outlier Dynamics in LLM NVFP4 Pretraining
by: Dong, Peijie, et al.
Published: (2026)

AdaMMS: Model Merging for Heterogeneous Multimodal Large Language Models with Unsupervised Coefficient Optimization
by: Du, Yiyang, et al.
Published: (2025)

Mediator: Memory-efficient LLM Merging with Less Parameter Conflicts and Uncertainty Based Routing
by: Lai, Kunfeng, et al.
Published: (2025)

GPAS: Accelerating Convergence of LLM Pretraining via Gradient-Preserving Activation Scaling
by: Chen, Tianhao, et al.
Published: (2025)

Optimal Brain Iterative Merging: Mitigating Interference in LLM Merging
by: Wang, Zhixiang, et al.
Published: (2025)

Split and Merge: Aligning Position Biases in LLM-based Evaluators
by: Li, Zongjie, et al.
Published: (2023)

Multi-Novelty: Improve the Diversity and Novelty of Contents Generated by Large Language Models via inference-time Multi-Views Brainstorming
by: Lagzian, Arash, et al.
Published: (2025)

SONIC: Segmented Optimized Nexus for Information Compression in Key-Value Caching
by: Chen, Hong, et al.
Published: (2026)

PonderLM-2: Pretraining LLM with Latent Thoughts in Continuous Space
by: Zeng, Boyi, et al.
Published: (2025)

Target-Oriented Pretraining Data Selection via Neuron-Activated Graph
by: Wang, Zijun, et al.
Published: (2026)