Cover Image

Completed Hyperparameter Transfer across Modules, Width, Depth, Batch and Duration

Fuente: arXiv

Saved in:

Bibliographic Details
Main Authors:	Mlodozeniec, Bruno, Ablin, Pierre, Béthune, Louis, Busbridge, Dan, Klein, Michal, Ramapuram, Jason, Cuturi, Marco
Format:	Preprint
Published:	2025
Subjects:	Machine Learning Artificial Intelligence
Online Access:	Acceder al recurso
Tags:	Add Tag No Tags, Be the first to tag this record!

Similar Items

The Geometries of Truth Are Orthogonal Across Tasks
by: Azizian, Waiss, et al.
Published: (2025)

Scaling Laws for Forgetting during Finetuning with Pretraining Data Injection
by: Bethune, Louis, et al.
Published: (2025)

Revisiting the Scaling Properties of Downstream Metrics in Large Language Model Training
by: Krajewski, Jakub, et al.
Published: (2025)

Distillation Scaling Laws
by: Busbridge, Dan, et al.
Published: (2025)

The Data-Quality Illusion: Rethinking Classifier-Based Quality Filtering for LLM Pretraining
by: Saada, Thiziri Nait, et al.
Published: (2025)

Poly-View Contrastive Learning
by: Shidani, Amitis, et al.
Published: (2024)

Multivariate Conformal Prediction using Optimal Transport
by: Klein, Michal, et al.
Published: (2025)

Learning Unmasking Policies for Diffusion Language Models
by: Jazbec, Metod, et al.
Published: (2025)

Nectar: Neural Estimation of Cached-Token Attention via Regression
by: Monteiro, João, et al.
Published: (2026)

A Small-Scale System for Autoregressive Program Synthesis Enabling Controlled Experimentation
by: Webb, Russ, et al.
Published: (2026)

Shielded Diffusion: Generating Novel and Diverse Images using Sparse Repellency
by: Kirchhof, Michael, et al.
Published: (2024)

DynaMiCS: Fine-tuning LLMs with Performance Constraints using Dynamic Mixtures
by: Gualdoni, Eleonora, et al.
Published: (2026)

Scaling Properties of Continuous Diffusion Spoken Language Models
by: Ramapuram, Jason, et al.
Published: (2026)

Stochastic KV Routing: Enabling Adaptive Depth-Wise Cache Sharing
by: Filippova, Anastasiia, et al.
Published: (2026)

Scaling Laws for Optimal Data Mixtures
by: Shukor, Mustafa, et al.
Published: (2025)

Scaling Categorical Flow Maps
by: Davis, Oscar, et al.
Published: (2026)

Sample and Map from a Single Convex Potential: Generation using Conjugate Moment Measures
by: Vesseron, Nina, et al.
Published: (2025)

SKATE, a Scalable Tournament Eval: Weaker LLMs differentiate between stronger ones using verifiable challenges
by: Gould, Dewi S. W., et al.
Published: (2025)

1-Lipschitz Neural Distance Fields
by: Coiffier, Guillaume, et al.
Published: (2024)

Beware of the Batch Size: Hyperparameter Bias in Evaluating LoRA
by: Lee, Sangyoon, et al.
Published: (2026)

LinEAS: End-to-end Learning of Activation Steering with a Distributional Loss
by: Rodriguez, Pau, et al.
Published: (2025)

Mix, Don't Tune: Bilingual Pre-Training Outperforms Hyperparameter Search in Data-Constrained Settings
by: Jeha, Paul, et al.
Published: (2026)

Understanding the Mechanisms of Fast Hyperparameter Transfer
by: Ghosh, Nikhil, et al.
Published: (2025)

Careful with that Scalpel: Improving Gradient Surgery with an EMA
by: Hsieh, Yu-Guan, et al.
Published: (2024)

Implicit meta-learning may lead language models to trust more reliable sources
by: Krasheninnikov, Dmitrii, et al.
Published: (2023)

Mixture of Universal Experts: Scaling Virtual Width via Depth-Width Transformation
by: Chen, Yilong, et al.
Published: (2026)

Hyperparameter Optimization of Constraint Programming Solvers
by: Haddad, Hedieh, et al.
Published: (2026)

The Design Space of Tri-Modal Masked Diffusion Models
by: Bethune, Louis, et al.
Published: (2026)

Locking Pretrained Weights via Deep Low-Rank Residual Distillation
by: Sakamoto, Keitaro, et al.
Published: (2026)

Learning Elastic Costs to Shape Monge Displacements
by: Klein, Michal, et al.
Published: (2023)

Influence Functions for Scalable Data Attribution in Diffusion Models
by: Mlodozeniec, Bruno, et al.
Published: (2024)

Dynamic-Width Speculative Beam Decoding for Efficient LLM Inference
by: Qin, Zongyue, et al.
Published: (2024)

Depth-Width tradeoffs in Algorithmic Reasoning of Graph Tasks with Transformers
by: Yehudai, Gilad, et al.
Published: (2025)

Theory, Analysis, and Best Practices for Sigmoid Self-Attention
by: Ramapuram, Jason, et al.
Published: (2024)

Parallel Hyperparameter Optimization Of Spiking Neural Network
by: Firmin, Thomas, et al.
Published: (2024)

Parameters vs FLOPs: Scaling Laws for Optimal Sparsity for Mixture-of-Experts Language Models
by: Abnar, Samira, et al.
Published: (2025)

Facial Width-to-Height Ratio Does Not Predict Self-Reported Behavioral Tendencies
by: Kosinski, Michal
Published: (2024)

Graph-Based Captioning: Enhancing Visual Descriptions by Interconnecting Region Captions
by: Hsieh, Yu-Guan, et al.
Published: (2024)

On the Infinite Width and Depth Limits of Predictive Coding Networks
by: Innocenti, Francesco, et al.
Published: (2026)

Distributional Training Data Attribution: What do Influence Functions Sample?
by: Mlodozeniec, Bruno, et al.
Published: (2025)