Image de couverture de livre

Dynamic Rank Reinforcement Learning for Adaptive Low-Rank Multi-Head Self Attention in Large Language Models

Fuente: arXiv

Enregistré dans:

Détails bibliographiques
Auteur principal:	Erden, Caner
Format:	Preprint
Publié:	2025
Sujets:	Machine Learning Computation and Language
Accès en ligne:	Acceder al recurso
Tags:	Ajouter un tag Pas de tags, Soyez le premier à ajouter un tag!

Documents similaires

Multiscale Aggregated Hierarchical Attention (MAHA): A Game Theoretic and Optimization Driven Approach to Efficient Contextual Modeling in Large Language Models
par: Erden, Caner
Publié: (2025)

Multi-Conditional Ranking with Large Language Models
par: Pezeshkpour, Pouya, et autres
Publié: (2024)

Adaptive Feature-based Low-Rank Compression of Large Language Models via Bayesian Optimization
par: Ji, Yixin, et autres
Publié: (2024)

BoRA: Bayesian Hierarchical Low-Rank Adaption for Multi-Task Large Language Models
par: Eide, Simen, et autres
Publié: (2024)

LCQ: Low-Rank Codebook based Quantization for Large Language Models
par: Cai, Wen-Pu, et autres
Publié: (2024)

QDyLoRA: Quantized Dynamic Low-Rank Adaptation for Efficient Large Language Model Tuning
par: Rajabzadeh, Hossein, et autres
Publié: (2024)

LoLCATs: On Low-Rank Linearizing of Large Language Models
par: Zhang, Michael, et autres
Publié: (2024)

FlexiGPT: Pruning and Extending Large Language Models with Low-Rank Weight Sharing
par: Smith, James Seale, et autres
Publié: (2025)

ResQ: Mixed-Precision Quantization of Large Language Models with Low-Rank Residuals
par: Saxena, Utkarsh, et autres
Publié: (2024)

A Bayesian Interpretation of Adaptive Low-Rank Adaptation
par: Chen, Haolin, et autres
Publié: (2024)

LoLA: Low-Rank Linear Attention With Sparse Caching
par: McDermott, Luke, et autres
Publié: (2025)

Towards Understanding the Nature of Attention with Low-Rank Sparse Decomposition
par: He, Zhengfu, et autres
Publié: (2025)

Found in the Middle: Permutation Self-Consistency Improves Listwise Ranking in Large Language Models
par: Tang, Raphael, et autres
Publié: (2023)

SlimSpec: Low-Rank Draft LM-Head for Accelerated Speculative Decoding
par: Plaksin, Anton, et autres
Publié: (2026)

Multiple Choice Learning of Low-Rank Adapters for Language Modeling
par: Letzelter, Victor, et autres
Publié: (2025)

Training-Free Bayesianization for Low-Rank Adapters of Large Language Models
par: Shi, Haizhou, et autres
Publié: (2024)

SoftLMs: Efficient Adaptive Low-Rank Approximation of Language Models using Soft-Thresholding Mechanism
par: Bhatnagar, Priyansh, et autres
Publié: (2024)

Accelerating the Low-Rank Decomposed Models
par: Hajimolahoseini, Habib, et autres
Publié: (2024)

SARA: Singular-Value Based Adaptive Low-Rank Adaption
par: Gu, Jihao, et autres
Publié: (2024)

Don't Pay Attention, PLANT It: Pretraining Attention via Learning-to-Rank
par: Roy, Debjyoti Saha, et autres
Publié: (2024)

BLoB: Bayesian Low-Rank Adaptation by Backpropagation for Large Language Models
par: Wang, Yibin, et autres
Publié: (2024)

Demystifying Low-Rank Knowledge Distillation in Large Language Models: Convergence, Generalization, and Information-Theoretic Guarantees
par: Soarez, Alberlucia Rafael, et autres
Publié: (2026)

Prompt-Dependent Ranking of Large Language Models with Uncertainty Quantification
par: Menendez, Angel Rodrigo Avelar, et autres
Publié: (2026)

Multi-Head Low-Rank Attention
par: Liu, Songtao, et autres
Publié: (2026)

Eigen Attention: Attention in Low-Rank Space for KV Cache Compression
par: Saxena, Utkarsh, et autres
Publié: (2024)

BA-LoRA: Bias-Alleviating Low-Rank Adaptation to Mitigate Catastrophic Inheritance in Large Language Models
par: Chang, Yupeng, et autres
Publié: (2024)

FLuRKA: Fast and accurate unified Low-Rank & Kernel Attention
par: Gupta, Ahan, et autres
Publié: (2023)

CoRA: Optimizing Low-Rank Adaptation with Common Subspace of Large Language Models
par: Xiao, Xiaojun, et autres
Publié: (2024)

Basis Selection: Low-Rank Decomposition of Pretrained Large Language Models for Target Applications
par: Li, Yang, et autres
Publié: (2024)

A3 : an Analytical Low-Rank Approximation Framework for Attention
par: Wong, Jeffrey T. H., et autres
Publié: (2025)

LoRA+: Efficient Low Rank Adaptation of Large Models
par: Hayou, Soufiane, et autres
Publié: (2024)

AFLoRA: Adaptive Freezing of Low Rank Adaptation in Parameter Efficient Fine-Tuning of Large Models
par: Liu, Zeyu, et autres
Publié: (2024)

GaLore$+$: Boosting Low-Rank Adaptation for LLMs with Cross-Head Projection
par: Liao, Xutao, et autres
Publié: (2024)

Sequences of Logits Reveal the Low Rank Structure of Language Models
par: Golowich, Noah, et autres
Publié: (2025)

Ranking Large Language Models without Ground Truth
par: Dhurandhar, Amit, et autres
Publié: (2024)

Improving Transformers with Dynamically Composable Multi-Head Attention
par: Xiao, Da, et autres
Publié: (2024)

Fast Forwarding Low-Rank Training
par: Rahamim, Adir, et autres
Publié: (2024)

MTL-LoRA: Low-Rank Adaptation for Multi-Task Learning
par: Yang, Yaming, et autres
Publié: (2024)

Unsupervised Contrast-Consistent Ranking with Language Models
par: Stoehr, Niklas, et autres
Publié: (2023)

GoRA: Gradient-driven Adaptive Low Rank Adaptation
par: He, Haonan, et autres
Publié: (2025)