Image de couverture de livre

TensorLLM: Tensorising Multi-Head Attention for Enhanced Reasoning and Compression in LLMs

Fuente: arXiv

Enregistré dans:

Détails bibliographiques
Auteurs principaux:	Gu, Yuxuan, Zhou, Wuyang, Iacovides, Giorgos, Mandic, Danilo
Format:	Preprint
Publié:	2025
Sujets:	Computation and Language Machine Learning
Accès en ligne:	Acceder al recurso
Tags:	Ajouter un tag Pas de tags, Soyez le premier à ajouter un tag!

Documents similaires

KromHC: Manifold-Constrained Hyper-Connections with Kronecker-Product Residual Matrices
par: Zhou, Wuyang, et autres
Publié: (2026)

FinDPO: Financial Sentiment Analysis for Algorithmic Trading through Preference Optimization of LLMs
par: Iacovides, Giorgos, et autres
Publié: (2025)

Towards LLM-guided Efficient and Interpretable Multi-linear Tensor Network Rank Selection
par: Iacovides, Giorgos, et autres
Publié: (2024)

Domain-Aware Tensor Network Structure Search
par: Iacovides, Giorgos, et autres
Publié: (2025)

TeRA: Vector-based Random Tensor Network for High-Rank Adaptation of Large Language Models
par: Gu, Yuxuan, et autres
Publié: (2025)

Understanding the Rank of Tensor Networks via an Intuitive Example-Driven Approach
par: Zhou, Wuyang, et autres
Publié: (2025)

RefineBridge: Generative Bridge Models Improve Financial Forecasting by Foundation Models
par: Bolton, Anthony, et autres
Publié: (2025)

FinLlama: Financial Sentiment Classification for Algorithmic Trading Applications
par: Konstantinidis, Thanos, et autres
Publié: (2024)

TensorGPT: Efficient Compression of Large Language Models based on Tensor-Train Decomposition
par: Xu, Mingxue, et autres
Publié: (2023)

Targeted Angular Reversal of Weights (TARS) for Knowledge Removal in Large Language Models
par: Davies, Harry J., et autres
Publié: (2024)

Geometry is All You Need: A Unified Taxonomy of Matrix and Tensor Factorization for Compression of Generative Language Models
par: Xu, Mingxue, et autres
Publié: (2024)

TensorSLM: Energy-efficient Embedding Compression of Sub-billion Parameter Language Models on Low-end Devices
par: Xu, Mingxue, et autres
Publié: (2025)

LatentLLM: Attention-Aware Joint Tensor Compression
par: Koike-Akino, Toshiaki, et autres
Publié: (2025)

RazorAttention: Efficient KV Cache Compression Through Retrieval Heads
par: Tang, Hanlin, et autres
Publié: (2024)

CHAI: Clustered Head Attention for Efficient LLM Inference
par: Agarwal, Saurabh, et autres
Publié: (2024)

Communication Compression for Tensor Parallel LLM Inference
par: Hansen-Palmus, Jan, et autres
Publié: (2024)

Improving Transformers with Dynamically Composable Multi-Head Attention
par: Xiao, Da, et autres
Publié: (2024)

Beyond KV Caching: Shared Attention for Efficient LLMs
par: Liao, Bingli, et autres
Publié: (2024)

Mechanism and Emergence of Stacked Attention Heads in Multi-Layer Transformers
par: Musat, Tiberiu
Publié: (2024)

Demystifying Long Chain-of-Thought Reasoning in LLMs
par: Yeo, Edward, et autres
Publié: (2025)

AAPO: Enhancing the Reasoning Capabilities of LLMs with Advantage Margin
par: Xiong, Jian, et autres
Publié: (2025)

Tensor Product Attention Is All You Need
par: Zhang, Yifan, et autres
Publié: (2025)

Enhancing the General Agent Capabilities of Low-Parameter LLMs through Tuning and Multi-Branch Reasoning
par: Zhou, Qinhao, et autres
Publié: (2024)

LLMSteer: Improving Long-Context LLM Inference by Steering Attention on Reused Contexts
par: Gu, Zhuohan, et autres
Publié: (2024)

LongLLMLingua: Accelerating and Enhancing LLMs in Long Context Scenarios via Prompt Compression
par: Jiang, Huiqiang, et autres
Publié: (2023)

HCAttention: Extreme KV Cache Compression via Heterogeneous Attention Computing for LLMs
par: Yang, Dongquan, et autres
Publié: (2025)

Multi-Head Attention Is a Multi-Player Game
par: Chakrabarti, Kushal, et autres
Publié: (2026)

LongFlow: Efficient KV Cache Compression for Reasoning Models
par: Su, Yi, et autres
Publié: (2026)

Dynamic Rank Reinforcement Learning for Adaptive Low-Rank Multi-Head Self Attention in Large Language Models
par: Erden, Caner
Publié: (2025)

Sparse Query Attention (SQA): A Computationally Efficient Attention Mechanism with Query Heads Reduction
par: Filipek, Adam
Publié: (2025)

ProxyAttn: Guided Sparse Attention via Representative Heads
par: Wang, Yixuan, et autres
Publié: (2025)

Unveiling Simplicities of Attention: Adaptive Long-Context Head Identification
par: Donhauser, Konstantin, et autres
Publié: (2025)

LoMA: Lossless Compressed Memory Attention
par: Wang, Yumeng, et autres
Publié: (2024)

Attention as a Compass: Efficient Exploration for Process-Supervised RL in Reasoning Models
par: Liu, Runze, et autres
Publié: (2025)

Self-playing Adversarial Language Game Enhances LLM Reasoning
par: Cheng, Pengyu, et autres
Publié: (2024)

Attention Illuminates LLM Reasoning: The Preplan-and-Anchor Rhythm Enables Fine-Grained Policy Optimization
par: Li, Yang, et autres
Publié: (2025)

Ltri-LLM: Streaming Long Context Inference for LLMs with Training-Free Dynamic Triangular Attention Pattern
par: Tang, Hongyin, et autres
Publié: (2024)

ALIEN: Aligned Entropy Head for Improving Uncertainty Estimation of LLMs
par: Zabolotnyi, Artem, et autres
Publié: (2025)

Training LLMs over Neurally Compressed Text
par: Lester, Brian, et autres
Publié: (2024)

From Debate to Equilibrium: Belief-Driven Multi-Agent LLM Reasoning via Bayesian Nash Equilibrium
par: Yi, Xie, et autres
Publié: (2025)