Initialization using Update Approximation is a Silver Bullet for Extremely Efficient Low-Rank Fine-Tuning

Fuente: arXiv

Saved in:

Bibliographic Details
Main Authors:	Ponkshe, Kaustubh, Singhal, Raghav, Gorbunov, Eduard, Tumanov, Alexey, Horvath, Samuel, Vepakomma, Praneeth
Format:	Preprint
Published:	2024
Subjects:	Computation and Language Artificial Intelligence Machine Learning
Online Access:	Acceder al recurso
Tags:	Add Tag No Tags, Be the first to tag this record!

Similar Items

FedEx-LoRA: Exact Aggregation for Federated and Efficient Fine-Tuning of Foundation Models
by: Singhal, Raghav, et al.
Published: (2024)

Fed-SB: A Silver Bullet for Extreme Communication Efficiency and Performance in (Private) Federated LoRA Fine-Tuning
by: Singhal, Raghav, et al.
Published: (2025)

ABBA-Adapters: Efficient and Expressive Fine-Tuning of Foundation Models
by: Singhal, Raghav, et al.
Published: (2025)

Safety Subspaces are Not Linearly Distinct: A Fine-Tuning Case Study
by: Ponkshe, Kaustubh, et al.
Published: (2025)

Power Mechanism: Private Tabular Representation Release for Model Agnostic Consumption
by: Vepakomma, Praneeth, et al.
Published: (2025)

Learning in the Null Space: Small Singular Values for Continual Learning
by: Pham, Cuong Anh, et al.
Published: (2026)

Low-Resource Machine Translation through the Lens of Personalized Federated Learning
by: Moskvoretskii, Viktor, et al.
Published: (2024)

SplitLoRA: A Split Parameter-Efficient Fine-Tuning Framework for Large Language Models
by: Lin, Zheng, et al.
Published: (2024)

Combinatorial Privacy: Private Multi-Party Bitstream Grand Sum by Hiding in Birkhoff Polytopes
by: Vepakomma, Praneeth
Published: (2026)

A Lightweight Method to Disrupt Memorized Sequences in LLM
by: Prashant, Parjanya Prajakta, et al.
Published: (2025)

MoRA: High-Rank Updating for Parameter-Efficient Fine-Tuning
by: Jiang, Ting, et al.
Published: (2024)

Localized LoRA: A Structured Low-Rank Approximation for Efficient Fine-Tuning
by: Barazandeh, Babak, et al.
Published: (2025)

StructFormer: Document Structure-based Masked Attention and its Impact on Language Model Pre-Training
by: Ponkshe, Kaustubh, et al.
Published: (2024)

Optimizing Fine-Tuning through Advanced Initialization Strategies for Low-Rank Adaptation
by: Xue, Yongfu
Published: (2025)

Modulated learning for private and distributed regression with just a single sample per client device
by: Vepakomma, Praneeth, et al.
Published: (2026)

DP-Fusion: Token-Level Differentially Private Inference for Large Language Models
by: Thareja, Rushil, et al.
Published: (2025)

Predicting Survival of Hemodialysis Patients using Federated Learning
by: Raju, Abhiram, et al.
Published: (2024)

GUIDEQ: Framework for Guided Questioning for progressive informational collection and classification
by: Mishra, Priya, et al.
Published: (2024)

Is Small Language Model the Silver Bullet to Low-Resource Languages Machine Translation?
by: Song, Yewei, et al.
Published: (2025)

LoRA$^2$ : Multi-Scale Low-Rank Approximations for Fine-Tuning Large Language Models
by: Zhang, Jia-Chen, et al.
Published: (2024)

Is Self-Repair a Silver Bullet for Code Generation?
by: Olausson, Theo X., et al.
Published: (2023)

DropLoRA: Sparse Low-Rank Adaptation for Parameter-Efficient Fine-Tuning
by: Zhang, Haojie
Published: (2025)

High-Rank Structured Modulation for Parameter-Efficient Fine-Tuning
by: Liu, Yongkang, et al.
Published: (2026)

TLoRA+: A Low-Rank Parameter-Efficient Fine-Tuning Method for Large Language Models
by: Cao, Yarui, et al.
Published: (2026)

TsqLoRA: Towards Sensitivity and Quality Low-Rank Adaptation for Efficient Fine-Tuning
by: Chen, Yu, et al.
Published: (2025)

AutoPureData: Automated Filtering of Undesirable Web Data to Update LLM Knowledge
by: Vadlapati, Praneeth
Published: (2024)

MELoRA: Mini-Ensemble Low-Rank Adapters for Parameter-Efficient Fine-Tuning
by: Ren, Pengjie, et al.
Published: (2024)

When Is Rank-1 Enough? Geometry-Guided Initialization for Parameter-Efficient Fine-Tuning
by: Zhao, Haoran, et al.
Published: (2026)

Beyond QA Pairs: Assessing Parameter-Efficient Fine-Tuning for Fact Embedding in LLMs
by: Ratnakar, Shivam, et al.
Published: (2025)

MSPLoRA: A Multi-Scale Pyramid Low-Rank Adaptation for Efficient Model Fine-Tuning
by: Zhao, Jiancheng, et al.
Published: (2025)

LoR2C : Low-Rank Residual Connection Adaptation for Parameter-Efficient Fine-Tuning
by: Zhao, Jiancheng, et al.
Published: (2025)

Rationales Are Not Silver Bullets: Measuring the Impact of Rationales on Model Performance and Reliability
by: Zhu, Chiwei, et al.
Published: (2025)

VB-LoRA: Extreme Parameter Efficient Fine-Tuning with Vector Banks
by: Li, Yang, et al.
Published: (2024)

Prompt, Translate, Fine-Tune, Re-Initialize, or Instruction-Tune? Adapting LLMs for In-Context Learning in Low-Resource Languages
by: Toukmaji, Christopher, et al.
Published: (2025)

LoFT: Low-Rank Adaptation That Behaves Like Full Fine-Tuning
by: Tastan, Nurbek, et al.
Published: (2025)

A Multi-Encoder Frozen-Decoder Approach for Fine-Tuning Large Language Models
by: Dhole, Kaustubh D.
Published: (2025)

FoRA: Fisher-orthogonal Rank Adaptation for Parameter-Efficient Fine-Tuning
by: Park, Juneyoung, et al.
Published: (2026)

DoRA: Enhancing Parameter-Efficient Fine-Tuning with Dynamic Rank Distribution
by: Mao, Yulong, et al.
Published: (2024)

AFLoRA: Adaptive Freezing of Low Rank Adaptation in Parameter Efficient Fine-Tuning of Large Models
by: Liu, Zeyu, et al.
Published: (2024)

Convergence of Clipped-SGD for Convex $(L_0,L_1)$-Smooth Optimization with Heavy-Tailed Noise
by: Chezhegov, Savelii, et al.
Published: (2025)