Cover Image

Head-wise Shareable Attention for Large Language Models

Fuente: arXiv

Saved in:

Bibliographic Details
Main Authors:	Cao, Zouying, Yang, Yifei, Zhao, Hai
Format:	Preprint
Published:	2024
Subjects:	Computation and Language
Online Access:	Acceder al recurso
Tags:	Add Tag No Tags, Be the first to tag this record!

Similar Items

LaCo: Large Language Model Pruning via Layer Collapse
by: Yang, Yifei, et al.
Published: (2024)

AutoHall: Automated Factuality Hallucination Dataset Generation for Large Language Models
by: Cao, Zouying, et al.
Published: (2023)

PGPO: Enhancing Agent Reasoning via Pseudocode-style Planning Guided Preference Optimization
by: Cao, Zouying, et al.
Published: (2025)

KVSharer: Efficient Inference via Layer-Wise Dissimilar KV Cache Sharing
by: Yang, Yifei, et al.
Published: (2024)

LESA: Learnable LLM Layer Scaling-Up
by: Yang, Yifei, et al.
Published: (2025)

Publicly Shareable Clinical Large Language Model Built on Synthetic Clinical Notes
by: Kweon, Sunjun, et al.
Published: (2023)

Attention Heads of Large Language Models: A Survey
by: Zheng, Zifan, et al.
Published: (2024)

SCANS: Mitigating the Exaggerated Safety for LLMs via Safety-Conscious Activation Steering
by: Cao, Zouying, et al.
Published: (2024)

EComStage: Stage-wise and Orientation-specific Benchmarking for Large Language Models in E-commerce
by: Zhao, Kaiyan, et al.
Published: (2026)

Remember Me, Refine Me: A Dynamic Procedural Memory Framework for Experience-Driven Agent Evolution
by: Cao, Zouying, et al.
Published: (2025)

On the Role of Attention Heads in Large Language Model Safety
by: Zhou, Zhenhong, et al.
Published: (2024)

Focusing on Language: Revealing and Exploiting Language Attention Heads in Multilingual Large Language Models
by: Liu, Xin, et al.
Published: (2025)

Garbage Attention in Large Language Models: BOS Sink Heads and Sink-aware Pruning
by: Sok, Jaewon, et al.
Published: (2026)

CoViPAL: Layer-wise Contextualized Visual Token Pruning for Large Vision-Language Models
by: Tang, Zicong, et al.
Published: (2025)

NoVo: Norm Voting off Hallucinations with Attention Heads in Large Language Models
by: Ho, Zheng Yi, et al.
Published: (2024)

Bias A-head? Analyzing Bias in Transformer-Based Language Model Attention Heads
by: Yang, Yi, et al.
Published: (2023)

Unitxt: Flexible, Shareable and Reusable Data Preparation and Evaluation for Generative AI
by: Bandel, Elron, et al.
Published: (2024)

Latent Multi-Head Attention for Small Language Models
by: Mehta, Sushant, et al.
Published: (2025)

LemmaHead: RAG Assisted Proof Generation Using Large Language Models
by: Yang, Tianbo, et al.
Published: (2025)

DESTEIN: Navigating Detoxification of Language Models via Universal Steering Pairs and Head-wise Activation Fusion
by: Li, Yu, et al.
Published: (2024)

Towards Comprehensive Stage-wise Benchmarking of Large Language Models in Fact-Checking
by: Lin, Hongzhan, et al.
Published: (2026)

Retrieval Backward Attention without Additional Training: Enhance Embeddings of Large Language Models via Repetition
by: Duan, Yifei, et al.
Published: (2025)

Learning to Route Queries to Heads for Attention-based Re-ranking with Large Language Models
by: Tian, Yuxing, et al.
Published: (2026)

SORSA: Singular Values and Orthonormal Regularized Singular Vectors Adaptation of Large Language Models
by: Cao, Yang, et al.
Published: (2024)

Token-wise Influential Training Data Retrieval for Large Language Models
by: Lin, Huawei, et al.
Published: (2024)

Knocking-Heads Attention
by: Zhou, Zhanchao, et al.
Published: (2025)

FTP: A Fine-grained Token-wise Pruner for Large Language Models via Token Routing
by: Li, Zekai, et al.
Published: (2024)

CharED: Character-wise Ensemble Decoding for Large Language Models
by: Gu, Kevin, et al.
Published: (2024)

Iterative Layer-wise Distillation for Efficient Compression of Large Language Models
by: Kovalev, Grigory, et al.
Published: (2025)

BriLLM: Brain-inspired Large Language Model
by: Zhao, Hai, et al.
Published: (2025)

Dynamic Rank Reinforcement Learning for Adaptive Low-Rank Multi-Head Self Attention in Large Language Models
by: Erden, Caner
Published: (2025)

MERIT: Maximum-normalized Element-wise Ratio for Language Model Large-batch Training
by: Luo, Yang, et al.
Published: (2025)

SAPT: A Shared Attention Framework for Parameter-Efficient Continual Learning of Large Language Models
by: Zhao, Weixiang, et al.
Published: (2024)

Layer-wise Regularized Dropout for Neural Language Models
by: Ni, Shiwen, et al.
Published: (2024)

On the Robustness of Editing Large Language Models
by: Ma, Xinbei, et al.
Published: (2024)

RRTL: Red Teaming Reasoning Large Language Models in Tool Learning
by: Liu, Yifei, et al.
Published: (2025)

Towards Inference-time Category-wise Safety Steering for Large Language Models
by: Bhattacharjee, Amrita, et al.
Published: (2024)

Gated Attention for Large Language Models: Non-linearity, Sparsity, and Attention-Sink-Free
by: Qiu, Zihan, et al.
Published: (2025)

Enhancing Large Language Models'Machine Translation via Dynamic Focus Anchoring
by: Ding, Qiuyu, et al.
Published: (2025)

RRAttention: Dynamic Block Sparse Attention via Per-Head Round-Robin Shifts for Long-Context Inference
by: Liu, Siran, et al.
Published: (2026)