Copertina

EG-MLA: Embedding-Gated Multi-head Latent Attention for Scalable and Efficient LLMs

Fuente: arXiv

Salvato in:

Dettagli Bibliografici
Autori principali:	Cai, Zhengge, Hou, Haowen
Natura:	Preprint
Pubblicazione:	2025
Soggetti:	Computation and Language
Accesso online:	Acceder al recurso
Tags:	Aggiungi Tag Nessun Tag, puoi essere il primo ad aggiungerne!!

Documenti analoghi

X-EcoMLA: Upcycling Pre-Trained Attention into MLA for Efficient and Extreme KV Compression
di: Li, Guihong, et al.
Pubblicazione: (2025)

SnapMLA: Efficient Long-Context MLA Decoding via Hardware-Aware FP8 Quantized Pipelining
di: Zhang, Yifan, et al.
Pubblicazione: (2026)

MHA2MLA-VLM: Enabling DeepSeek's Economical Multi-Head Latent Attention across Vision-Language Models
di: Fan, Xiaoran, et al.
Pubblicazione: (2026)

EmbeddingRWKV: State-Centric Retrieval with Reusable States
di: Hou, Haowen, et al.
Pubblicazione: (2026)

In-context KV-Cache Eviction for LLMs via Attention-Gate
di: Zeng, Zihao, et al.
Pubblicazione: (2024)

Explicit Multi-head Attention for Inter-head Interaction in Large Language Models
di: Peng, Runyu, et al.
Pubblicazione: (2026)

Gated Slot Attention for Efficient Linear-Time Sequence Modeling
di: Zhang, Yu, et al.
Pubblicazione: (2024)

Interleaved Latent Visual Reasoning with Selective Perceptual Modeling
di: Dong, Shuai, et al.
Pubblicazione: (2025)

Advancing Sentiment Analysis: A Novel LSTM Framework with Multi-head Attention
di: Yi, Jingyuan, et al.
Pubblicazione: (2025)

Pair-In, Pair-Out: Latent Multi-Token Prediction for Efficient LLMs
di: Tan, Wenhui, et al.
Pubblicazione: (2026)

Gated Tree Cross-Attention for Checkpoint-Compatible Syntax Injection in Decoder-Only LLMs
di: Gao, Xinyu, et al.
Pubblicazione: (2026)

Gated Linear Attention Transformers with Hardware-Efficient Training
di: Yang, Songlin, et al.
Pubblicazione: (2023)

Efficient Ternary Weight Embedding Model: Bridging Scalability and Performance
di: Chen, Jiayi, et al.
Pubblicazione: (2024)

MARRO: Multi-headed Attention for Rhetorical Role Labeling in Legal Documents
di: Bambroo, Purbid, et al.
Pubblicazione: (2025)

OpenGuardrails: A Configurable, Unified, and Scalable Guardrails Platform for Large Language Models
di: Wang, Thomas, et al.
Pubblicazione: (2025)

Compressed Convolutional Attention: Efficient Attention in a Compressed Latent Space
di: Figliolia, Tomas, et al.
Pubblicazione: (2025)

CausalEmbed: Auto-Regressive Multi-Vector Generation in Latent Space for Visual Document Embedding
di: Huo, Jiahao, et al.
Pubblicazione: (2026)

GTA: Grouped-head latenT Attention
di: Sun, Luoyang, et al.
Pubblicazione: (2025)

Optimizing Native Sparse Attention with Latent Attention and Local Global Alternating Strategies
di: Hu, Yuxuan, et al.
Pubblicazione: (2025)

Arctic-Embed: Scalable, Efficient, and Accurate Text Embedding Models
di: Merrick, Luke, et al.
Pubblicazione: (2024)

VisualRWKV-HD and UHD: Advancing High-Resolution Processing for Visual Language Models
di: Li, Zihang, et al.
Pubblicazione: (2024)

Towards Economical Inference: Enabling DeepSeek's Multi-Head Latent Attention in Any Transformer-based LLMs
di: Ji, Tao, et al.
Pubblicazione: (2025)

Do Multilingual LLMs have specialized language heads?
di: Naufil, Muhammad
Pubblicazione: (2026)

Latent Multi-Head Attention for Small Language Models
di: Mehta, Sushant, et al.
Pubblicazione: (2025)

Towards Better Multi-head Attention via Channel-wise Sample Permutation
di: Yuan, Shen, et al.
Pubblicazione: (2024)

ReGLA: Refining Gated Linear Attention
di: Lu, Peng, et al.
Pubblicazione: (2025)

Griffin: Mixing Gated Linear Recurrences with Local Attention for Efficient Language Models
di: De, Soham, et al.
Pubblicazione: (2024)

Fast-MIA: Efficient and Scalable Membership Inference for LLMs
di: Takahashi, Hiromu, et al.
Pubblicazione: (2025)

Flux Attention: Context-Aware Hybrid Attention for Efficient LLMs Inference
di: Qiu, Quantong, et al.
Pubblicazione: (2026)

FlashMLA-ETAP: Efficient Transpose Attention Pipeline for Accelerating MLA Inference on NVIDIA H20 GPUs
di: Dege, Pengcuo, et al.
Pubblicazione: (2025)

Uncertainty-Aware Attention Heads: Efficient Unsupervised Uncertainty Quantification for LLMs
di: Vazhentsev, Artem, et al.
Pubblicazione: (2025)

Attn-GS: Attention-Guided Context Compression for Efficient Personalized LLMs
di: Zeng, Shenglai, et al.
Pubblicazione: (2026)

Answer-Centric or Reasoning-Driven? Uncovering the Latent Memory Anchor in LLMs
di: Wu, Yang, et al.
Pubblicazione: (2025)

Gated Attention for Large Language Models: Non-linearity, Sparsity, and Attention-Sink-Free
di: Qiu, Zihan, et al.
Pubblicazione: (2025)

SR-KI: Scalable and Real-Time Knowledge Integration into LLMs via Supervised Attention
di: Yu, Bohan, et al.
Pubblicazione: (2025)

GAProtoNet: A Multi-head Graph Attention-based Prototypical Network for Interpretable Text Classification
di: Wen, Ximing, et al.
Pubblicazione: (2024)

Scalable Multi-phase Word Embedding Using Conjunctive Propositional Clauses
di: Kadhim, Ahmed K., et al.
Pubblicazione: (2025)

Learning to Look at the Other Side: A Semantic Probing Study of Word Embeddings in LLMs with Enabled Bidirectional Attention
di: Feng, Zhaoxin, et al.
Pubblicazione: (2025)

Efficient Latent Semantic Clustering for Scaling Test-Time Computation of LLMs
di: Lee, Sungjae, et al.
Pubblicazione: (2025)

Bias A-head? Analyzing Bias in Transformer-Based Language Model Attention Heads
di: Yang, Yi, et al.
Pubblicazione: (2023)