Cover Image

High-dimensional Analysis of Knowledge Distillation: Weak-to-Strong Generalization and Scaling Laws

Fuente: arXiv

Saved in:

Bibliographic Details
Main Authors:	Ildiz, M. Emrullah, Gozeten, Halil Alperen, Taga, Ege Onur, Mondelli, Marco, Oymak, Samet
Format:	Preprint
Published:	2024
Subjects:	Machine Learning
Online Access:	Acceder al recurso
Tags:	Add Tag No Tags, Be the first to tag this record!

Similar Items

Learning to Correct: Calibrated Reinforcement Learning for Multi-Attempt Chain-of-Thought
by: Ildiz, Muhammed Emrullah, et al.
Published: (2026)

Evolutionary Multi-Task Optimization for LLM-Guided Program Discovery
by: Gozeten, Halil Alperen, et al.
Published: (2026)

TimePFN: Effective Multivariate Time Series Forecasting with Synthetic Data
by: Taga, Ege Onur, et al.
Published: (2025)

Test-Time Training Provably Improves Transformers as In-context Learners
by: Gozeten, Halil Alperen, et al.
Published: (2025)

Retrieval Augmented Time Series Forecasting
by: Tire, Kutay, et al.
Published: (2024)

Continuous Chain of Thought Enables Parallel Exploration and Reasoning
by: Gozeten, Halil Alperen, et al.
Published: (2025)

Learning to Bet for Horizon-Aware Anytime-Valid Testing
by: Taga, Ege Onur, et al.
Published: (2026)

Covariance-Aware Transformers for Quadratic Programming and Decision Making
by: Tire, Kutay, et al.
Published: (2026)

From Self-Attention to Markov Models: Unveiling the Dynamics of Generative Transformers
by: Ildiz, M. Emrullah, et al.
Published: (2024)

Mechanics of Next Token Prediction with Self-Attention
by: Li, Yingcong, et al.
Published: (2024)

Improved Scaling Laws via Weak-to-Strong Generalization in Random Feature Ridge Regression
by: Wu, Diyuan, et al.
Published: (2026)

Attention with Trained Embeddings Provably Selects Important Tokens
by: Wu, Diyuan, et al.
Published: (2025)

Efficient Contextual LLM Cascades through Budget-Constrained Policy Learning
by: Zhang, Xuechen, et al.
Published: (2024)

On the Generalization Properties of Selective State-Space Models for Filtering Tasks for Unknown Systems
by: Tang, Alex, et al.
Published: (2026)

Latent Chain-of-Thought Improves Structured-Data Transformers
by: Dudley, Carson, et al.
Published: (2026)

High-dimensional Analysis of Synthetic Data Selection
by: Rezaei, Parham, et al.
Published: (2025)

Universal Lower Bounds and Optimal Rates: Achieving Minimax Clustering Error in Sub-Exponential Mixture Models
by: Dreveton, Maximilien, et al.
Published: (2024)

Fine-grained Analysis of In-context Linear Estimation: Data, Architecture, and Beyond
by: Li, Yingcong, et al.
Published: (2024)

How Spurious Features Are Memorized: Precise Analysis for Random and NTK Features
by: Bombari, Simone, et al.
Published: (2023)

Spurious Correlations in High Dimensional Regression: The Roles of Regularization, Simplicity Bias and Over-Parameterization
by: Bombari, Simone, et al.
Published: (2025)

Neural Collapse Beyond the Unconstrained Features Model: Landscape, Dynamics, and Generalization in the Mean-Field Regime
by: Wu, Diyuan, et al.
Published: (2025)

A Law of Data Reconstruction for Random Features (and Beyond)
by: Iurada, Leonardo, et al.
Published: (2025)

In-Context Learning Under Regime Change
by: Dudley, Carson, et al.
Published: (2026)

Spectral Estimators for Multi-Index Models: Precise Asymptotics and Optimal Weak Recovery
by: Kovačević, Filip, et al.
Published: (2025)

Can Transformers Learn Optimal Filtering for Unknown Systems?
by: Balim, Haldun, et al.
Published: (2023)

Optimal Representation Size: High-Dimensional Analysis of Pretraining and Linear Probing
by: Njaradi, Valentina, et al.
Published: (2026)

Learning Rate Should Scale Inversely with High-Order Data Moments in High-Dimensional Online Independent Component Analysis
by: Gultekin, M. Oguzhan, et al.
Published: (2025)

On the Power of Convolution Augmented Transformer
by: Li, Mingchen, et al.
Published: (2024)

Privacy for Free in the Overparameterized Regime
by: Bombari, Simone, et al.
Published: (2024)

Towards Understanding the Word Sensitivity of Attention Layers: A Study via Random Features
by: Bombari, Simone, et al.
Published: (2024)

High-Dimensional Private Linear Regression with Optimal Rates
by: Bombari, Simone, et al.
Published: (2025)

Class-attribute Priors: Adapting Optimization to Heterogeneity and Fairness Objective
by: Zhang, Xuechen, et al.
Published: (2024)

Theoretical Analysis of Weak-to-Strong Generalization
by: Lang, Hunter, et al.
Published: (2024)

BREAD: Branched Rollouts from Expert Anchors Bridge SFT & RL for Reasoning
by: Zhang, Xuechen, et al.
Published: (2025)

Making Small Language Models Efficient Reasoners: Intervention, Supervision, Reinforcement
by: Zhang, Xuechen, et al.
Published: (2025)

The Mechanism of Weak-to-Strong Generalization: Feature Elicitation from Latent Knowledge
by: Awano, Ryoya, et al.
Published: (2026)

Precise Asymptotics for Spectral Methods in Mixed Generalized Linear Models
by: Zhang, Yihan, et al.
Published: (2022)

Transformers as Support Vector Machines
by: Tarzanagh, Davoud Ataee, et al.
Published: (2023)

Identifying Trustworthiness Challenges in Deep Learning Models for Continental-Scale Water Quality Prediction
by: Xia, Xiaobo, et al.
Published: (2025)

Optimal Regularization for Performative Learning
by: Cyffers, Edwige, et al.
Published: (2025)