Share Your Attention: Transformer Weight Sharing via Matrix-based Dictionary Learning

Fuente: arXiv

Saved in:

Bibliographic Details
Main Authors:	Zhussip, Magauiya, Shopkhoev, Dmitriy, Ali, Ammar, Lefkimmiatis, Stamatios
Format:	Preprint
Published:	2025
Subjects:	Computation and Language Artificial Intelligence Machine Learning
Online Access:	Acceder al recurso
Tags:	Add Tag No Tags, Be the first to tag this record!

Similar Items

CoSpaDi: Compressing LLMs via Calibration-Guided Sparse Dictionary Learning
by: Makhov, Denis, et al.
Published: (2025)

ROCKET: Rapid Optimization via Calibration-guided Knapsack Enhanced Truncation for Efficient Model Compression
by: Ali, Ammar, et al.
Published: (2026)

ReplaceMe: Network Simplification via Depth Pruning and Transformer Block Linearization
by: Shopkhoev, Dmitriy, et al.
Published: (2025)

Speak, Edit, Repeat: High-Fidelity Voice Editing and Zero-Shot TTS with Cross-Attentive Mamba
by: Mohammad, Baher, et al.
Published: (2025)

COMPOT: Calibration-Optimized Matrix Procrustes Orthogonalization for Transformers Compression
by: Makhov, Denis, et al.
Published: (2026)

A Modular Conditional Diffusion Framework for Image Reconstruction
by: Zhussip, Magauiya, et al.
Published: (2024)

Beyond KV Caching: Shared Attention for Efficient LLMs
by: Liao, Bingli, et al.
Published: (2024)

Advancing Arabic Reverse Dictionary Systems: A Transformer-Based Approach with Dataset Construction Guidelines
by: Sibaee, Serry, et al.
Published: (2025)

Check Your LLM's Secret Dictionary! Five Lines of Code Reveal What Your LLM Learned (Including What It Shouldn't Have)
by: Miyashita, Hisashi
Published: (2026)

Experience Sharing in Mutual Reinforcement Learning for Heterogeneous Language Models
by: Liu, Xiaoze, et al.
Published: (2026)

Accelerating Prefilling for Long-Context LLMs via Sparse Pattern Sharing
by: Peng, Dan, et al.
Published: (2025)

KVSharer: Efficient Inference via Layer-Wise Dissimilar KV Cache Sharing
by: Yang, Yifei, et al.
Published: (2024)

Your Transformer is Secretly Linear
by: Razzhigaev, Anton, et al.
Published: (2024)

How Sparse Attention Approximates Exact Attention? Your Attention is Naturally $n^C$-Sparse
by: Deng, Yichuan, et al.
Published: (2024)

MoSLD: An Extremely Parameter-Efficient Mixture-of-Shared LoRAs for Multi-Task Learning
by: Zhao, Lulu, et al.
Published: (2024)

Shared Lexical Task Representations Explain Behavioral Variability In LLMs
by: Yang, Zhuonan, et al.
Published: (2026)

Mixture of Attentions For Speculative Decoding
by: Zimmer, Matthieu, et al.
Published: (2024)

More Expressive Attention with Negative Weights
by: Lv, Ang, et al.
Published: (2024)

OMPILOT: Harnessing Transformer Models for Auto Parallelization to Shared Memory Computing Paradigms
by: Bhattacharjee, Arijit, et al.
Published: (2025)

Rethinking Parameter Sharing for LLM Fine-Tuning with Multiple LoRAs
by: Ban, Hao, et al.
Published: (2025)

DHA: Learning Decoupled-Head Attention from Transformer Checkpoints via Adaptive Heads Fusion
by: Chen, Yilong, et al.
Published: (2024)

Features that Make a Difference: Leveraging Gradients for Improved Dictionary Learning
by: Olmo, Jeffrey, et al.
Published: (2024)

Shared Doubt: Zero-shot Cross-Lingual Confidence Estimation for Language Models
by: Kyriakou, Athina, et al.
Published: (2026)

Towards Interpretable Sequence Continuation: Analyzing Shared Circuits in Large Language Models
by: Lan, Michael, et al.
Published: (2023)

Measuring Progress in Dictionary Learning for Language Model Interpretability with Board Game Models
by: Karvonen, Adam, et al.
Published: (2024)

In-Context Learning with Transformers: Softmax Attention Adapts to Function Lipschitzness
by: Collins, Liam, et al.
Published: (2024)

How Private is Your Attention? Bridging Privacy with In-Context Learning
by: Bonnerjee, Soham, et al.
Published: (2025)

Subliminal Effects in Your Data: A General Mechanism via Log-Linearity
by: Aden-Ali, Ishaq, et al.
Published: (2026)

Untangling Component Imbalance in Hybrid Linear Attention Conversion Methods
by: Benfeghoul, Martin, et al.
Published: (2025)

Selective Attention Improves Transformer
by: Leviathan, Yaniv, et al.
Published: (2024)

JoMA: Demystifying Multilayer Transformers via JOint Dynamics of MLP and Attention
by: Tian, Yuandong, et al.
Published: (2023)

Beyond Linear Approximations: A Novel Pruning Approach for Attention Matrix
by: Liang, Yingyu, et al.
Published: (2024)

Hierarchical vs. Flat Iteration in Shared-Weight Transformers
by: Han, Sang-Il
Published: (2026)

Enhancing Hyperspace Analogue to Language (HAL) Representations via Attention-Based Pooling for Text Classification
by: Sakour, Ali, et al.
Published: (2026)

Low-Cost Generation and Evaluation of Dictionary Example Sentences
by: Cai, Bill, et al.
Published: (2024)

Attention Mechanisms Don't Learn Additive Models: Rethinking Feature Importance for Transformers
by: Leemann, Tobias, et al.
Published: (2024)

GSQ-Tuning: Group-Shared Exponents Integer in Fully Quantized Training for LLMs On-Device Fine-tuning
by: Zhou, Sifan, et al.
Published: (2025)

KUET at StanceNakba Shared Task: StanceMoE: Mixture-of-Experts Architecture for Stance Detection
by: Shafi, Abdullah Al, et al.
Published: (2026)

What Matters in Transformers? Not All Attention is Needed
by: He, Shwai, et al.
Published: (2024)

Not All Synthetic Data Is Yours to Learn From
by: Alemohammad, Sina, et al.
Published: (2026)