Cover Image

Distillation Scaling Laws

Fuente: arXiv

Saved in:

Bibliographic Details
Main Authors:	Busbridge, Dan, Shidani, Amitis, Weers, Floris, Ramapuram, Jason, Littwin, Etai, Webb, Russ
Format:	Preprint
Published:	2025
Subjects:	Machine Learning Artificial Intelligence Computation and Language
Online Access:	Acceder al recurso
Tags:	Add Tag No Tags, Be the first to tag this record!

Similar Items

Revisiting the Scaling Properties of Downstream Metrics in Large Language Model Training
by: Krajewski, Jakub, et al.
Published: (2025)

Scaling Properties of Continuous Diffusion Spoken Language Models
by: Ramapuram, Jason, et al.
Published: (2026)

Poly-View Contrastive Learning
by: Shidani, Amitis, et al.
Published: (2024)

Theory, Analysis, and Best Practices for Sigmoid Self-Attention
by: Ramapuram, Jason, et al.
Published: (2024)

UI-JEPA: Towards Active Perception of User Intent through Onscreen User Activity
by: Fu, Yicheng, et al.
Published: (2024)

When can transformers reason with abstract symbols?
by: Boix-Adsera, Enric, et al.
Published: (2023)

Vanishing Gradients in Reinforcement Finetuning of Language Models
by: Razin, Noam, et al.
Published: (2023)

A Small-Scale System for Autoregressive Program Synthesis Enabling Controlled Experimentation
by: Webb, Russ, et al.
Published: (2026)

Completed Hyperparameter Transfer across Modules, Width, Depth, Batch and Duration
by: Mlodozeniec, Bruno, et al.
Published: (2025)

MaskTab: Scalable Masked Tabular Pretraining with Scaling Laws and Distillation for Industrial Classification
by: Zheng, Bo, et al.
Published: (2026)

Scaling Laws for Forgetting during Finetuning with Pretraining Data Injection
by: Bethune, Louis, et al.
Published: (2025)

Towards the Law of Capacity Gap in Distilling Language Models
by: Zhang, Chen, et al.
Published: (2023)

What Scales in Cross-Entropy Scaling Law?
by: Yan, Junxi, et al.
Published: (2025)

Scaling Law with Learning Rate Annealing
by: Tissue, Howe, et al.
Published: (2024)

P$^2$ Law: Scaling Law for Post-Training After Model Pruning
by: Chen, Xiaodong, et al.
Published: (2024)

Parameters vs FLOPs: Scaling Laws for Optimal Sparsity for Mixture-of-Experts Language Models
by: Abnar, Samira, et al.
Published: (2025)

Can Language Models Discover Scaling Laws?
by: Lin, Haowei, et al.
Published: (2025)

Exploring Scaling Laws for EHR Foundation Models
by: Zhang, Sheng, et al.
Published: (2025)

Theoretical Foundations of Scaling Law in Familial Models
by: Song, Huan, et al.
Published: (2025)

Scaling Laws for Predicting Downstream Performance in LLMs
by: Chen, Yangyi, et al.
Published: (2024)

Scaling Laws for Fine-Grained Mixture of Experts
by: Krajewski, Jakub, et al.
Published: (2024)

A Hitchhiker's Guide to Scaling Law Estimation
by: Choshen, Leshem, et al.
Published: (2024)

Towards a Comprehensive Scaling Law of Mixture-of-Experts
by: Zhao, Guoliang, et al.
Published: (2025)

Predicting Task Performance with Context-aware Scaling Laws
by: Montgomery, Kyle, et al.
Published: (2025)

Relative-Based Scaling Law for Neural Language Models
by: Yue, Baoqing, et al.
Published: (2025)

Unifying Learning Dynamics and Generalization in Transformers Scaling Law
by: Yang, Chiwun
Published: (2025)

Loss-to-Loss Prediction: Scaling Laws for All Datasets
by: Brandfonbrener, David, et al.
Published: (2024)

Observational Scaling Laws and the Predictability of Language Model Performance
by: Ruan, Yangjun, et al.
Published: (2024)

To Memorize or to Retrieve: Scaling Laws for RAG-Considerate Pretraining
by: Singh, Karan, et al.
Published: (2026)

The Power of Power Law: Asymmetry Enables Compositional Reasoning
by: Wang, Zixuan, et al.
Published: (2026)

Distilling Multi-Scale Knowledge for Event Temporal Relation Extraction
by: Yao, Hao-Ren, et al.
Published: (2022)

LLMs on the Line: Data Determines Loss-to-Loss Scaling Laws
by: Mayilvahanan, Prasanna, et al.
Published: (2025)

Diversity of Transformer Layers: One Aspect of Parameter Scaling Laws
by: Kamigaito, Hidetaka, et al.
Published: (2025)

Scaling Laws for Reward Model Overoptimization in Direct Alignment Algorithms
by: Rafailov, Rafael, et al.
Published: (2024)

Uncovering Scaling Laws for Large Language Models via Inverse Problems
by: Verma, Arun, et al.
Published: (2025)

Provable Scaling Laws for the Test-Time Compute of Large Language Models
by: Chen, Yanxi, et al.
Published: (2024)

Batched Contextual Reinforcement: A Task-Scaling Law for Efficient Reasoning
by: Yang, Bangji, et al.
Published: (2026)

A Tale of Tails: Model Collapse as a Change of Scaling Laws
by: Dohmatob, Elvis, et al.
Published: (2024)

(Mis)Fitting: A Survey of Scaling Laws
by: Li, Margaret, et al.
Published: (2025)

Joint MoE Scaling Laws: Mixture of Experts Can Be Memory Efficient
by: Ludziejewski, Jan, et al.
Published: (2025)