Cover Image

The Magic Correlations: Understanding Knowledge Transfer from Pretraining to Supervised Fine-Tuning

Fuente: arXiv

Saved in:

Bibliographic Details
Main Authors:	Fan, Simin, Paparas, Dimitris, Noy, Natasha, Xiong, Binbin, Sachdeva, Noveen, Isik, Berivan
Format:	Preprint
Published:	2026
Subjects:	Machine Learning
Online Access:	Acceder al recurso
Tags:	Add Tag No Tags, Be the first to tag this record!

Similar Items

Scaling Laws for Downstream Task Performance of Large Language Models
by: Isik, Berivan, et al.
Published: (2024)

ReMem: Mutual Information-Aware Fine-tuning of Pretrained Vision Transformers for Effective Knowledge Distillation
by: Dong, Chengyu, et al.
Published: (2025)

GC4NC: A Benchmark Framework for Graph Condensation on Node Classification with New Insights
by: Gong, Shengbo, et al.
Published: (2024)

Differentially Private Adaptation of Diffusion Models via Noisy Aggregated Embeddings
by: Peetathawatchai, Pura, et al.
Published: (2024)

GRAPE: Optimize Data Mixture for Group Robust Multi-target Adaptive Pretraining
by: Fan, Simin, et al.
Published: (2025)

TuneShift-KD: Knowledge Distillation and Transfer for Fine-tuned Models
by: Guan, Yushi, et al.
Published: (2026)

Not Every AI Problem is a Data Problem: We Should Be Intentional About Data Scaling
by: Rodchenko, Tanya, et al.
Published: (2025)

Successive Halving with Learning Curve Prediction via Latent Kronecker Gaussian Processes
by: Lin, Jihao Andreas, et al.
Published: (2025)

Improved Communication-Privacy Trade-offs in $L_2$ Mean Estimation under Streaming Differential Privacy
by: Chen, Wei-Ning, et al.
Published: (2024)

On Fairness of Low-Rank Adaptation of Large Models
by: Ding, Zhoujie, et al.
Published: (2024)

Anchored Supervised Fine-Tuning
by: Zhu, He, et al.
Published: (2025)

Efficient Hyperparameter Search for Non-Stationary Model Training
by: Isik, Berivan, et al.
Published: (2025)

BiSSL: Enhancing the Alignment Between Self-Supervised Pretraining and Downstream Fine-Tuning via Bilevel Optimization
by: Zakarias, Gustav Wagner, et al.
Published: (2024)

Understanding Task Transfer in Vision-Language Models
by: Sachdeva, Bhuvan, et al.
Published: (2025)

Task-Adaptive Pretrained Language Models via Clustered-Importance Sampling
by: Grangier, David, et al.
Published: (2024)

A Theory of How Pretraining Shapes Inductive Bias in Fine-Tuning
by: Anguita, Nicolas, et al.
Published: (2026)

Label-Efficient School Detection from Aerial Imagery via Weakly Supervised Pretraining and Fine-Tuning
by: Elmimouni, Zakarya, et al.
Published: (2026)

Edge-Assisted Collaborative Fine-Tuning for Multi-User Personalized Artificial Intelligence Generated Content (AIGC)
by: Li, Nan, et al.
Published: (2025)

Adaptive Compression in Federated Learning via Side Information
by: Isik, Berivan, et al.
Published: (2023)

Rotation-Preserving Supervised Fine-Tuning
by: Jin, Hangzhan, et al.
Published: (2026)

Simplifying Knowledge Transfer in Pretrained Models
by: Jain, Siddharth, et al.
Published: (2025)

Proximal Supervised Fine-Tuning
by: Zhu, Wenhong, et al.
Published: (2025)

Robust Fine-Tuning from Non-Robust Pretrained Models: Mitigating Suboptimal Transfer With Epsilon-Scheduling
by: Ngnawé, Jonas, et al.
Published: (2025)

Portable Reward Tuning: Towards Reusable Fine-Tuning across Different Pretrained Models
by: Chijiwa, Daiki, et al.
Published: (2025)

Relationships are Complicated! An Analysis of Relationships Between Datasets on the Web
by: Lin, Kate, et al.
Published: (2024)

Kernel Banzhaf: A Fast and Robust Estimator for Banzhaf Values
by: Liu, Yurong, et al.
Published: (2024)

UFT: Unifying Supervised and Reinforcement Fine-Tuning
by: Liu, Mingyang, et al.
Published: (2025)

Diversity in Large Language Models under Supervised Fine-Tuning
by: Klypa, Roman, et al.
Published: (2026)

Closed-Loop Supervised Fine-Tuning of Tokenized Traffic Models
by: Zhang, Zhejun, et al.
Published: (2024)

Unsupervised Identification and Removal of Spurious Correlations During Fine-Tuning
by: Gilligan-Lee, Ciarán M., et al.
Published: (2026)

Scaling Pretrained Representations Enables Label-Free Out-of-Distribution Detection Without Fine-Tuning
by: Barkley, Brett, et al.
Published: (2026)

GenFT: A Generative Parameter-Efficient Fine-Tuning Method for Pretrained Foundation Models
by: Zhang, Baoquan, et al.
Published: (2025)

Transfer Learning in Physics-Informed Neural Networks: Full Fine-Tuning, Lightweight Fine-Tuning, and Low-Rank Adaptation
by: Wang, Yizheng, et al.
Published: (2025)

Unlock the Correlation between Supervised Fine-Tuning and Reinforcement Learning in Training Code Large Language Models
by: Chen, Jie, et al.
Published: (2024)

A Layer-wise Analysis of Supervised Fine-Tuning
by: Zhao, Qinghua, et al.
Published: (2026)

Goal-Conditioned Supervised Learning for LLM Fine-Tuning
by: Li, Shijun, et al.
Published: (2026)

Erasing the Bias: Fine-Tuning Foundation Models for Semi-Supervised Learning
by: Gan, Kai, et al.
Published: (2024)

Supervised Fine-Tuning Achieve Rapid Task Adaption Via Alternating Attention Head Activation Patterns
by: Zhao, Yang, et al.
Published: (2024)

Optimizing DDPM Sampling with Shortcut Fine-Tuning
by: Fan, Ying, et al.
Published: (2023)

Understanding and Preserving Safety in Fine-Tuned LLMs
by: Zhang, Jiawen, et al.
Published: (2026)