Imagen de Portada

Theoretical Insights into Fine-Tuning Attention Mechanism: Generalization and Optimization

Fuente: arXiv

Guardado en:

Detalles Bibliográficos
Autores principales:	Yao, Xinhao, Qian, Hongjin, Hu, Xiaolin, Xu, Gengze, Liu, Wei, Luan, Jian, Wang, Bin, Liu, Yong
Formato:	Preprint
Publicado:	2024
Materias:	Machine Learning
Acceso en línea:	Acceder al recurso
Etiquetas:	Agregar Etiqueta Sin Etiquetas, Sea el primero en etiquetar este registro!

Ejemplares similares

On the Emergence of Weak-to-Strong Generalization: A Bias-Variance Perspective
por: Xu, Gengze, et al.
Publicado: (2025)

Enhancing In-Context Learning Performance with just SVD-Based Weight Pruning: A Theoretical Perspective
por: Yao, Xinhao, et al.
Publicado: (2024)

PMSS: Pretrained Matrices Skeleton Selection for LLM Fine-tuning
por: Wang, Qibin, et al.
Publicado: (2024)

On the Blessing of Pre-training in Weak-to-Strong Generalization
por: Yao, Wei, et al.
Publicado: (2026)

The Capabilities and Limitations of Weak-to-Strong Generalization: Generalization and Calibration
por: Yao, Wei, et al.
Publicado: (2025)

DoTA: Weight-Decomposed Tensor Adaptation for Large Language Models
por: Hu, Xiaolin, et al.
Publicado: (2024)

On Weak-to-Strong Generalization and f-Divergence
por: Yao, Wei, et al.
Publicado: (2025)

Compositional Generalization from Learned Skills via CoT Training: A Theoretical and Structural Analysis for Reasoning
por: Yao, Xinhao, et al.
Publicado: (2025)

Look Within or Look Beyond? A Theoretical Comparison Between Parameter-Efficient and Full Fine-Tuning
por: Liu, Yongkang, et al.
Publicado: (2025)

The Debate on RLVR Reasoning Capability Boundary: Shrinkage, Expansion, or Both? A Two-Stage Dynamic View
por: Yao, Xinhao, et al.
Publicado: (2025)

Enhancing Reinforcement Learning Fine-Tuning with an Online Refiner
por: Ma, Hao, et al.
Publicado: (2026)

Control Theoretic Approach to Fine-Tuning and Transfer Learning
por: Bayram, Erkan, et al.
Publicado: (2024)

LoSiA: Efficient High-Rank Fine-Tuning via Subnet Localization and Optimization
por: Wang, Xujia, et al.
Publicado: (2025)

Multi-branch of Attention Yields Accurate Results for Tabular Data
por: Li, Xuechen, et al.
Publicado: (2025)

PSEO: Optimizing Post-hoc Stacking Ensemble Through Hyperparameter Tuning
por: Xu, Beicheng, et al.
Publicado: (2025)

SASA: Semantic-Aware Contrastive Learning Framework with Separated Attention for Triple Classification
por: Xiaodan, Xu, et al.
Publicado: (2026)

Self-Generative Adversarial Fine-Tuning for Large Language Models
por: Wu, Shiguang, et al.
Publicado: (2026)

Beyond Progress Measures: Theoretical Insights into the Mechanism of Grokking
por: Gu, Zihan, et al.
Publicado: (2025)

Learning to Think: Information-Theoretic Reinforcement Fine-Tuning for LLMs
por: Wang, Jingyao, et al.
Publicado: (2025)

Attention Mechanism, Max-Affine Partition, and Universal Approximation
por: Liu, Hude, et al.
Publicado: (2025)

Fine-Tuning Without Forgetting In-Context Learning: A Theoretical Analysis of Linear Attention Models
por: Lee, Chungpa, et al.
Publicado: (2026)

Fed-pilot: Optimizing LoRA Allocation for Efficient Federated Fine-Tuning with Heterogeneous Clients
por: Zhang, Zikai, et al.
Publicado: (2024)

ChunkFT: Byte-Streamed Optimization for Memory-Efficient Full Fine-Tuning
por: Liu, Yongkang, et al.
Publicado: (2026)

Towards Understanding Fine-Tuning Mechanisms of LLMs via Circuit Analysis
por: Wang, Xu, et al.
Publicado: (2025)

Towards Auto-Regressive Next-Token Prediction: In-Context Learning Emerges from Generalization
por: Gong, Zixuan, et al.
Publicado: (2025)

Understanding and Preserving Safety in Fine-Tuned LLMs
por: Zhang, Jiawen, et al.
Publicado: (2026)

Mixture of Diverse Size Experts
por: Sun, Manxi, et al.
Publicado: (2024)

Bi-LoRA: Efficient Sharpness-Aware Minimization for Fine-Tuning Large-Scale Models
por: Liu, Yuhang, et al.
Publicado: (2025)

Information-Theoretic Generalization Bounds for Transductive Learning and its Applications
por: Tang, Huayi, et al.
Publicado: (2023)

Graph Attention is Not Always Beneficial: A Theoretical Analysis of Graph Attention Mechanisms via Contextual Stochastic Block Models
por: Ma, Zhongtian, et al.
Publicado: (2024)

Generative Representational Instruction Tuning
por: Muennighoff, Niklas, et al.
Publicado: (2024)

Preserving Domain Generalization in Fine-Tuning via Joint Parameter Selection
por: Pan, Bin, et al.
Publicado: (2025)

Towards a Theoretical Understanding to the Generalization of RLHF
por: Li, Zhaochun, et al.
Publicado: (2026)

An Optimization Framework for Differentially Private Sparse Fine-Tuning
por: Makni, Mehdi, et al.
Publicado: (2025)

Invariance Makes LLM Unlearning Resilient Even to Unanticipated Downstream Fine-Tuning
por: Wang, Changsheng, et al.
Publicado: (2025)

LoRA vs. Full Fine-Tuning: A Theoretical Perspective
por: Zindari, Ali, et al.
Publicado: (2026)

RPO:Reinforcement Fine-Tuning with Partial Reasoning Optimization
por: Yi, Hongzhu, et al.
Publicado: (2026)

Adaptive and Fine-grained Module-wise Expert Pruning for Efficient LoRA-MoE Fine-Tuning
por: Li, Weihang, et al.
Publicado: (2026)

Parameter-Efficient Fine-Tuning for Large Models: A Comprehensive Survey
por: Han, Zeyu, et al.
Publicado: (2024)

RedRFT: A Light-Weight Benchmark for Reinforcement Fine-Tuning-Based Red Teaming
por: Zheng, Xiang, et al.
Publicado: (2025)