Cover Image

Dataset distillation for memorized data: Soft labels can leak held-out teacher knowledge

Fuente: arXiv

Saved in:

Bibliographic Details
Main Authors:	Behrens, Freya, Zdeborová, Lenka
Format:	Preprint
Published:	2025
Subjects:	Machine Learning
Online Access:	Acceder al recurso
Tags:	Add Tag No Tags, Be the first to tag this record!

Similar Items

Counting in Small Transformers: The Delicate Interplay between Attention and Feed-Forward Layers
by: Behrens, Freya, et al.
Published: (2024)

A phase transition between positional and semantic learning in a solvable model of dot-product attention
by: Cui, Hugo, et al.
Published: (2024)

Dynamical Cavity Method for Hypergraphs and its Application to Quenches in the k-XOR-SAT Problem
by: Maier, Aude, et al.
Published: (2024)

Counting and Hardness-of-Finding Fixed Points in Cellular Automata on Random Graphs
by: Koller, Cédric, et al.
Published: (2024)

Dynamical Phase Transitions in Graph Cellular Automata
by: Behrens, Freya, et al.
Published: (2023)

High-dimensional Asymptotics of Denoising Autoencoders
by: Cui, Hugo, et al.
Published: (2023)

Building Conformal Prediction Intervals with Approximate Message Passing
by: Clarté, Lucas, et al.
Published: (2024)

Optimal thresholds and algorithms for a model of multi-modal learning in high dimensions
by: Keup, Christian, et al.
Published: (2024)

Spectral Thresholds in Correlated Spiked Models and Fundamental Limits of Partial Least Squares
by: Mergny, Pierre, et al.
Published: (2025)

Gibbs Sampling the Posterior of Neural Networks
by: Piccioli, Giovanni, et al.
Published: (2023)

Trust the uncertain teacher: distilling dark knowledge via calibrated uncertainty
by: Kim, Jeonghyun, et al.
Published: (2026)

Learning with Restricted Boltzmann Machines: Asymptotics of AMP and GD in High Dimensions
by: Xu, Yizhou, et al.
Published: (2025)

The Rules-and-Facts Model for Simultaneous Generalization and Memorization in Neural Networks
by: Farné, Gabriele, et al.
Published: (2026)

The Computational Advantage of Depth: Learning High-Dimensional Hierarchical Functions with Gradient Descent
by: Dandi, Yatin, et al.
Published: (2025)

Fundamental limits of Non-Linear Low-Rank Matrix Estimation
by: Mergny, Pierre, et al.
Published: (2024)

Minority Takeover in Majority Dynamics: Searching for Rare Initializations via the History Passing Algorithm
by: Jankola, Marek, et al.
Published: (2025)

Quenches in the Sherrington-Kirkpatrick model
by: Erba, Vittorio, et al.
Published: (2024)

On the existence of consistent adversarial attacks in high-dimensional linear classification
by: Vilucchio, Matteo, et al.
Published: (2025)

Analysis of learning a flow-based generative model from limited sample complexity
by: Cui, Hugo, et al.
Published: (2023)

Inference in Spreading Processes with Neural-Network Priors
by: Ghio, Davide, et al.
Published: (2025)

Computational Thresholds in Multi-Modal Learning via the Spiked Matrix-Tensor Model
by: Tabanelli, Hugo, et al.
Published: (2025)

Exploring the potential of prototype-based soft-labels data distillation for imbalanced data classification
by: Rosu, Radu-Andrei, et al.
Published: (2024)

Rigorous Asymptotics for First-Order Algorithms Through the Dynamical Cavity Method
by: Dandi, Yatin, et al.
Published: (2026)

Bayes optimal learning of attention-indexed models
by: Boncoraglio, Fabrizio, et al.
Published: (2025)

The Nuclear Route: Sharp Asymptotics of ERM in Overparameterized Quadratic Networks
by: Erba, Vittorio, et al.
Published: (2025)

Sampling with flows, diffusion and autoregressive neural networks: A spin-glass perspective
by: Ghio, Davide, et al.
Published: (2023)

Universality laws for Gaussian mixtures in generalized linear models
by: Dandi, Yatin, et al.
Published: (2023)

On student-teacher deviations in distillation: does it pay to disobey?
by: Nagarajan, Vaishnavh, et al.
Published: (2023)

Fundamental Limits of Matrix Sensing: Exact Asymptotics, Universality, and Applications
by: Xu, Yizhou, et al.
Published: (2025)

The Benefits of Reusing Batches for Gradient Descent in Two-Layer Networks: Breaking the Curse of Information and Leap Exponents
by: Dandi, Yatin, et al.
Published: (2024)

Asymptotics of SGD in Sequence-Single Index Models and Single-Layer Attention Networks
by: Arnaboldi, Luca, et al.
Published: (2025)

Fundamental limits of learning in sequence multi-index models and deep attention networks: High-dimensional asymptotics and sharp thresholds
by: Troiani, Emanuele, et al.
Published: (2025)

Bilinear Sequence Regression: A Model for Learning from Long Sequences of High-dimensional Tokens
by: Erba, Vittorio, et al.
Published: (2024)

Rigorous dynamical mean field theory for stochastic gradient descent methods
by: Gerbelot, Cedric, et al.
Published: (2022)

KD$^{2}$M: A unifying framework for feature knowledge distillation
by: Montesuma, Eduardo Fernandes
Published: (2025)

Extraction of linearized models from pre-trained networks via knowledge distillation
by: Kimura, Fumito, et al.
Published: (2026)

What is the role of memorization in Continual Learning?
by: Kozal, Jędrzej, et al.
Published: (2025)

Scalability of memorization-based machine unlearning
by: Zhao, Kairan, et al.
Published: (2024)

Gaussian Universality of Perceptrons with Random Labels
by: Gerace, Federica, et al.
Published: (2022)

Bayes-optimal learning of an extensive-width neural network from quadratically many samples
by: Maillard, Antoine, et al.
Published: (2024)