Copertina

Dynamic Rank Reinforcement Learning for Adaptive Low-Rank Multi-Head Self Attention in Large Language Models

Fuente: arXiv

Salvato in:

Dettagli Bibliografici
Autore principale:	Erden, Caner
Natura:	Preprint
Pubblicazione:	2025
Soggetti:	Machine Learning Computation and Language
Accesso online:	Acceder al recurso
Tags:	Aggiungi Tag Nessun Tag, puoi essere il primo ad aggiungerne!!

Documenti analoghi

Multiscale Aggregated Hierarchical Attention (MAHA): A Game Theoretic and Optimization Driven Approach to Efficient Contextual Modeling in Large Language Models
di: Erden, Caner
Pubblicazione: (2025)

Multi-Conditional Ranking with Large Language Models
di: Pezeshkpour, Pouya, et al.
Pubblicazione: (2024)

Adaptive Feature-based Low-Rank Compression of Large Language Models via Bayesian Optimization
di: Ji, Yixin, et al.
Pubblicazione: (2024)

BoRA: Bayesian Hierarchical Low-Rank Adaption for Multi-Task Large Language Models
di: Eide, Simen, et al.
Pubblicazione: (2024)

LCQ: Low-Rank Codebook based Quantization for Large Language Models
di: Cai, Wen-Pu, et al.
Pubblicazione: (2024)

QDyLoRA: Quantized Dynamic Low-Rank Adaptation for Efficient Large Language Model Tuning
di: Rajabzadeh, Hossein, et al.
Pubblicazione: (2024)

LoLCATs: On Low-Rank Linearizing of Large Language Models
di: Zhang, Michael, et al.
Pubblicazione: (2024)

FlexiGPT: Pruning and Extending Large Language Models with Low-Rank Weight Sharing
di: Smith, James Seale, et al.
Pubblicazione: (2025)

ResQ: Mixed-Precision Quantization of Large Language Models with Low-Rank Residuals
di: Saxena, Utkarsh, et al.
Pubblicazione: (2024)

A Bayesian Interpretation of Adaptive Low-Rank Adaptation
di: Chen, Haolin, et al.
Pubblicazione: (2024)

LoLA: Low-Rank Linear Attention With Sparse Caching
di: McDermott, Luke, et al.
Pubblicazione: (2025)

Towards Understanding the Nature of Attention with Low-Rank Sparse Decomposition
di: He, Zhengfu, et al.
Pubblicazione: (2025)

Found in the Middle: Permutation Self-Consistency Improves Listwise Ranking in Large Language Models
di: Tang, Raphael, et al.
Pubblicazione: (2023)

SlimSpec: Low-Rank Draft LM-Head for Accelerated Speculative Decoding
di: Plaksin, Anton, et al.
Pubblicazione: (2026)

Multiple Choice Learning of Low-Rank Adapters for Language Modeling
di: Letzelter, Victor, et al.
Pubblicazione: (2025)

Training-Free Bayesianization for Low-Rank Adapters of Large Language Models
di: Shi, Haizhou, et al.
Pubblicazione: (2024)

SoftLMs: Efficient Adaptive Low-Rank Approximation of Language Models using Soft-Thresholding Mechanism
di: Bhatnagar, Priyansh, et al.
Pubblicazione: (2024)

Accelerating the Low-Rank Decomposed Models
di: Hajimolahoseini, Habib, et al.
Pubblicazione: (2024)

SARA: Singular-Value Based Adaptive Low-Rank Adaption
di: Gu, Jihao, et al.
Pubblicazione: (2024)

Don't Pay Attention, PLANT It: Pretraining Attention via Learning-to-Rank
di: Roy, Debjyoti Saha, et al.
Pubblicazione: (2024)

BLoB: Bayesian Low-Rank Adaptation by Backpropagation for Large Language Models
di: Wang, Yibin, et al.
Pubblicazione: (2024)

Demystifying Low-Rank Knowledge Distillation in Large Language Models: Convergence, Generalization, and Information-Theoretic Guarantees
di: Soarez, Alberlucia Rafael, et al.
Pubblicazione: (2026)

Prompt-Dependent Ranking of Large Language Models with Uncertainty Quantification
di: Menendez, Angel Rodrigo Avelar, et al.
Pubblicazione: (2026)

Multi-Head Low-Rank Attention
di: Liu, Songtao, et al.
Pubblicazione: (2026)

Eigen Attention: Attention in Low-Rank Space for KV Cache Compression
di: Saxena, Utkarsh, et al.
Pubblicazione: (2024)

BA-LoRA: Bias-Alleviating Low-Rank Adaptation to Mitigate Catastrophic Inheritance in Large Language Models
di: Chang, Yupeng, et al.
Pubblicazione: (2024)

FLuRKA: Fast and accurate unified Low-Rank & Kernel Attention
di: Gupta, Ahan, et al.
Pubblicazione: (2023)

CoRA: Optimizing Low-Rank Adaptation with Common Subspace of Large Language Models
di: Xiao, Xiaojun, et al.
Pubblicazione: (2024)

Basis Selection: Low-Rank Decomposition of Pretrained Large Language Models for Target Applications
di: Li, Yang, et al.
Pubblicazione: (2024)

A3 : an Analytical Low-Rank Approximation Framework for Attention
di: Wong, Jeffrey T. H., et al.
Pubblicazione: (2025)

LoRA+: Efficient Low Rank Adaptation of Large Models
di: Hayou, Soufiane, et al.
Pubblicazione: (2024)

AFLoRA: Adaptive Freezing of Low Rank Adaptation in Parameter Efficient Fine-Tuning of Large Models
di: Liu, Zeyu, et al.
Pubblicazione: (2024)

GaLore$+$: Boosting Low-Rank Adaptation for LLMs with Cross-Head Projection
di: Liao, Xutao, et al.
Pubblicazione: (2024)

Sequences of Logits Reveal the Low Rank Structure of Language Models
di: Golowich, Noah, et al.
Pubblicazione: (2025)

Ranking Large Language Models without Ground Truth
di: Dhurandhar, Amit, et al.
Pubblicazione: (2024)

Improving Transformers with Dynamically Composable Multi-Head Attention
di: Xiao, Da, et al.
Pubblicazione: (2024)

Fast Forwarding Low-Rank Training
di: Rahamim, Adir, et al.
Pubblicazione: (2024)

MTL-LoRA: Low-Rank Adaptation for Multi-Task Learning
di: Yang, Yaming, et al.
Pubblicazione: (2024)

Unsupervised Contrast-Consistent Ranking with Language Models
di: Stoehr, Niklas, et al.
Pubblicazione: (2023)

GoRA: Gradient-driven Adaptive Low Rank Adaptation
di: He, Haonan, et al.
Pubblicazione: (2025)