Cover Image

Variance Control via Weight Rescaling in LLM Pre-training

Fuente: arXiv

Saved in:

Bibliographic Details
Main Authors:	Owen, Louis, Kumar, Abhay, Chowdhury, Nilabhra Roy, Güra, Fabian
Format:	Preprint
Published:	2025
Subjects:	Machine Learning Computation and Language
Online Access:	Acceder al recurso
Tags:	Add Tag No Tags, Be the first to tag this record!

Similar Items

ZClip: Adaptive Spike Mitigation for LLM Pre-Training
by: Kumar, Abhay, et al.
Published: (2025)

A Refined Analysis of Massive Activations in LLMs
by: Owen, Louis, et al.
Published: (2025)

Power Lines: Scaling Laws for Weight Decay and Batch Size in LLM Pre-training
by: Bergsma, Shane, et al.
Published: (2025)

High-Layer Attention Pruning with Rescaling
by: Liu, Songtao, et al.
Published: (2025)

nanoLM: an Affordable LLM Pre-training Benchmark via Accurate Loss Prediction across Scales
by: Yao, Yiqun, et al.
Published: (2023)

Pre-training LLM without Learning Rate Decay Enhances Supervised Fine-Tuning
by: Yano, Kazuki, et al.
Published: (2026)

NuNER: Entity Recognition Encoder Pre-training via LLM-Annotated Data
by: Bogdanov, Sergei, et al.
Published: (2024)

Thinking Augmented Pre-training
by: Wang, Liang, et al.
Published: (2025)

Efficient Knowledge Probing of Large Language Models by Adapting Pre-trained Embeddings
by: Sharma, Kartik, et al.
Published: (2025)

Structural Pruning of Pre-trained Language Models via Neural Architecture Search
by: Klein, Aaron, et al.
Published: (2024)

STAR: Spectral Truncation and Rescale for Model Merging
by: Lee, Yu-Ang, et al.
Published: (2025)

Bootstrapping Post-training Signals for Open-ended Tasks via Rubric-based Self-play on Pre-training Text
by: Huang, Chengyu, et al.
Published: (2026)

Parameter Efficient Fine-tuning via Explained Variance Adaptation
by: Paischer, Fabian, et al.
Published: (2024)

On Limitations of LLM as Annotator for Low Resource Languages
by: Jadhav, Suramya, et al.
Published: (2024)

WSM: Decay-Free Learning Rate Schedule via Checkpoint Merging for LLM Pre-training
by: Tian, Changxin, et al.
Published: (2025)

Detecting Token-Level Hallucinations Using Variance Signals: A Reference-Free Approach
by: Kumar, Keshav
Published: (2025)

Model Merging in Pre-training of Large Language Models
by: Li, Yunshui, et al.
Published: (2025)

DEPT: Decoupled Embeddings for Pre-training Language Models
by: Iacob, Alex, et al.
Published: (2024)

Tackling Length Inflation Without Trade-offs: Group Relative Reward Rescaling for Reinforcement Learning
by: Li, Zichao, et al.
Published: (2026)

RVPO: Risk-Sensitive Alignment via Variance Regularization
by: Montero, Ivan, et al.
Published: (2026)

SpacTor-T5: Pre-training T5 Models with Span Corruption and Replaced Token Detection
by: Ye, Ke, et al.
Published: (2024)

Unmasking Backdoors: An Explainable Defense via Gradient-Attention Anomaly Scoring for Pre-trained Language Models
by: Das, Anindya Sundar, et al.
Published: (2025)

MM1: Methods, Analysis & Insights from Multimodal LLM Pre-training
by: McKinzie, Brandon, et al.
Published: (2024)

Selective Attention: Enhancing Transformer through Principled Context Control
by: Zhang, Xuechen, et al.
Published: (2024)

Using Pre-trained LLMs for Multivariate Time Series Forecasting
by: Wolff, Malcolm L., et al.
Published: (2025)

Exploiting Vocabulary Frequency Imbalance in Language Model Pre-training
by: Chung, Woojin, et al.
Published: (2025)

Efficient Continual Pre-training of LLMs for Low-resource Languages
by: Nag, Arijit, et al.
Published: (2024)

Making Pre-trained Language Models Great on Tabular Prediction
by: Yan, Jiahuan, et al.
Published: (2024)

The Dark Side of the Language: Pre-trained Transformers in the DarkNet
by: Ranaldi, Leonardo, et al.
Published: (2022)

TraceNAS: Zero-shot LLM Pruning via Gradient Trace Correlation
by: Malettira, Prajna G., et al.
Published: (2026)

Methods of improving LLM training stability
by: Rybakov, Oleg, et al.
Published: (2024)

PaPaformer: Language Model from Pre-trained Parallel Paths
by: Tapaninaho, Joonas, et al.
Published: (2025)

Learn or Recall? Revisiting Incremental Learning with Pre-trained Language Models
by: Zheng, Junhao, et al.
Published: (2023)

MediSwift: Efficient Sparse Pre-trained Biomedical Language Models
by: Thangarasa, Vithursan, et al.
Published: (2024)

PolarQuant: Optimal Gaussian Weight Quantization via Hadamard Rotation for LLM Compression
by: Vicentino, Caio
Published: (2026)

Hallucination Detection via Activations of Open-Weight Proxy Analyzers
by: Singh, Akshita, et al.
Published: (2026)

SWSC: Shared Weight for Similar Channel in LLM
by: Zeng, Binrui, et al.
Published: (2025)

Demystifying Synthetic Data in LLM Pre-training: A Systematic Study of Scaling Laws, Benefits, and Pitfalls
by: Kang, Feiyang, et al.
Published: (2025)

WISCA: A Lightweight Model Transition Method to Improve LLM Training via Weight Scaling
by: Li, Jiacheng, et al.
Published: (2025)

Fine-Tuning Pre-trained Language Models to Detect In-Game Trash Talks
by: Fesalbon, Daniel, et al.
Published: (2024)