Cover Image

Retrieval Head Mechanistically Explains Long-Context Factuality

Fuente: arXiv

Saved in:

Bibliographic Details
Main Authors:	Wu, Wenhao, Wang, Yizhong, Xiao, Guangxuan, Peng, Hao, Fu, Yao
Format:	Preprint
Published:	2024
Subjects:	Computation and Language
Online Access:	Acceder al recurso
Tags:	Add Tag No Tags, Be the first to tag this record!

Similar Items

DuoAttention: Efficient Long-Context LLM Inference with Retrieval and Streaming Heads
by: Xiao, Guangxuan, et al.
Published: (2024)

Long Context Alignment with Short Instructions and Synthesized Positions
by: Wu, Wenhao, et al.
Published: (2024)

Context-Efficient Retrieval with Factual Decomposition
by: Li, Yanhong, et al.
Published: (2025)

LongEmbed: Extending Embedding Models for Long Context Retrieval
by: Zhu, Dawei, et al.
Published: (2024)

Quest: Query-Aware Sparsity for Efficient Long-Context LLM Inference
by: Tang, Jiaming, et al.
Published: (2024)

Induction Head Toxicity Mechanistically Explains Repetition Curse in Large Language Models
by: Wang, Shuxun, et al.
Published: (2025)

InfLLM: Training-Free Long-Context Extrapolation for LLMs with an Efficient Context Memory
by: Xiao, Chaojun, et al.
Published: (2024)

ZigzagAttention: Efficient Long-Context Inference with Exclusive Retrieval and Streaming Heads
by: Liu, Zhuorui, et al.
Published: (2025)

From Interpretability to Performance: Optimizing Retrieval Heads for Long-Context Language Models
by: Ma, Youmi, et al.
Published: (2026)

Query-Focused Retrieval Heads Improve Long-Context Reasoning and Re-ranking
by: Zhang, Wuwei, et al.
Published: (2025)

VERA: Identifying and Leveraging Visual Evidence Retrieval Heads in Long-Context Understanding
by: Pei, Rongcan, et al.
Published: (2026)

FlashBack:Efficient Retrieval-Augmented Language Modeling for Long Context Inference
by: Liu, Runheng, et al.
Published: (2024)

Investigating the Factual Knowledge Boundary of Large Language Models with Retrieval Augmentation
by: Ren, Ruiyang, et al.
Published: (2023)

Understanding Synthetic Context Extension via Retrieval Heads
by: Zhao, Xinyu, et al.
Published: (2024)

HGOT: Hierarchical Graph of Thoughts for Retrieval-Augmented In-Context Learning in Factuality Evaluation
by: Fang, Yihao, et al.
Published: (2024)

Long$^2$RAG: Evaluating Long-Context & Long-Form Retrieval-Augmented Generation with Key Point Recall
by: Qi, Zehan, et al.
Published: (2024)

LongHeads: Multi-Head Attention is Secretly a Long Context Processor
by: Lu, Yi, et al.
Published: (2024)

Mechanistic Understanding and Mitigation of Language Model Non-Factual Hallucinations
by: Yu, Lei, et al.
Published: (2024)

Co-occurrence is not Factual Association in Language Models
by: Zhang, Xiao, et al.
Published: (2024)

Knowledgeable In-Context Tuning: Exploring and Exploiting Factual Knowledge for In-Context Learning
by: Wang, Jianing, et al.
Published: (2023)

FoRAG: Factuality-optimized Retrieval Augmented Generation for Web-enhanced Long-form Question Answering
by: Cai, Tianchi, et al.
Published: (2024)

Infinite Retrieval: Attention Enhanced LLMs in Long-Context Processing
by: Ye, Xiaoju, et al.
Published: (2025)

Inference Scaling for Long-Context Retrieval Augmented Generation
by: Yue, Zhenrui, et al.
Published: (2024)

Factuality and Transparency Are All RAG Needs! Self-Explaining Contrastive Evidence Re-ranking
by: Vargas, Francielle, et al.
Published: (2025)

Mindscape-Aware Retrieval Augmented Generation for Improved Long Context Understanding
by: Li, Yuqing, et al.
Published: (2025)

CTkvr: KV Cache Retrieval for Long-Context LLMs via Centroid then Token Indexing
by: Lu, Kuan, et al.
Published: (2025)

A Little Goes a Long Way: Efficient Long Context Training and Inference with Partial Contexts
by: Ge, Suyu, et al.
Published: (2024)

Only Say What You Know: Calibration-Aware Generation for Long-Form Factuality
by: Luo, Wen, et al.
Published: (2026)

Beyond Factual Accuracy: Evaluating Coverage of Diverse Factual Information in Long-form Text Generation
by: Samarinas, Chris, et al.
Published: (2025)

Locret: Enhancing Eviction in Long-Context LLM Inference with Trained Retaining Heads on Consumer-Grade Devices
by: Huang, Yuxiang, et al.
Published: (2024)

Agent-as-Judge for Factual Summarization of Long Narratives
by: Jeong, Yeonseok, et al.
Published: (2025)

RRAttention: Dynamic Block Sparse Attention via Per-Head Round-Robin Shifts for Long-Context Inference
by: Liu, Siran, et al.
Published: (2026)

Not All Heads Matter: A Head-Level KV Cache Compression Method with Integrated Retrieval and Reasoning
by: Fu, Yu, et al.
Published: (2024)

Collapse of Dense Retrievers: Short, Early, and Literal Biases Outranking Factual Evidence
by: Fayyaz, Mohsen, et al.
Published: (2025)

Challenges in Deploying Long-Context Transformers: A Theoretical Peak Performance Analysis
by: Fu, Yao
Published: (2024)

WhenLoss: Diagnosing Write and Retrieval Bottlenecks in Long-Context Memory Systems
by: Yu, Jiangnan, et al.
Published: (2026)

Is Factuality Enhancement a Free Lunch For LLMs? Better Factuality Can Lead to Worse Context-Faithfulness
by: Bi, Baolong, et al.
Published: (2024)

Efficient Prompt Compression with Evaluator Heads for Long-Context Transformer Inference
by: Fei, Weizhi, et al.
Published: (2025)

BTR: Binary Token Representations for Efficient Retrieval Augmented Language Models
by: Cao, Qingqing, et al.
Published: (2023)

S2-Attention: Hardware-Aware Context Sharding Among Attention Heads
by: Lin, Xihui, et al.
Published: (2024)