Cover Image

Understanding Synthetic Context Extension via Retrieval Heads

Fuente: arXiv

Saved in:

Bibliographic Details
Main Authors:	Zhao, Xinyu, Yin, Fangcong, Durrett, Greg
Format:	Preprint
Published:	2024
Subjects:	Computation and Language
Online Access:	Acceder al recurso
Tags:	Add Tag No Tags, Be the first to tag this record!

Similar Items

LoFiT: Localized Fine-tuning on LLM Representations
by: Yin, Fangcong, et al.
Published: (2024)

Query-Focused Retrieval Heads Improve Long-Context Reasoning and Re-ranking
by: Zhang, Wuwei, et al.
Published: (2025)

Learning Composable Chains-of-Thought
by: Yin, Fangcong, et al.
Published: (2025)

LongProc: Benchmarking Long-Context Language Models on Long Procedural Generation
by: Ye, Xi, et al.
Published: (2025)

SynthesizRR: Generating Diverse Datasets with Retrieval Augmentation
by: Divekar, Abhishek, et al.
Published: (2024)

Detecting and Suppressing Reward Hacking with Gradient Fingerprints
by: Wang, Songtao, et al.
Published: (2026)

Learning to Refine with Fine-Grained Natural Language Feedback
by: Wadhwa, Manya, et al.
Published: (2024)

To CoT or not to CoT? Chain-of-thought helps mainly on math and symbolic reasoning
by: Sprague, Zayne, et al.
Published: (2024)

Molecular Facts: Desiderata for Decontextualization in LLM Fact Verification
by: Gunjal, Anisha, et al.
Published: (2024)

Complex Claim Verification with Evidence Retrieved in the Wild
by: Chen, Jifan, et al.
Published: (2023)

Contrastive Learning to Improve Retrieval for Real-world Fact Checking
by: Sriram, Aniruddh, et al.
Published: (2024)

DySCO: Dynamic Attention-Scaling Decoding for Long-Context Language Models
by: Ye, Xi, et al.
Published: (2026)

X-PARADE: Cross-Lingual Textual Entailment and Information Divergence across Paragraphs
by: Rodriguez, Juan Diego, et al.
Published: (2023)

MiniCheck: Efficient Fact-Checking of LLMs on Grounding Documents
by: Tang, Liyan, et al.
Published: (2024)

From Distributional to Overton Pluralism: Investigating Large Language Model Alignment
by: Lake, Thom, et al.
Published: (2024)

Calibrate-Then-Act: Cost-Aware Exploration in LLM Agents
by: Ding, Wenxuan, et al.
Published: (2026)

Adaptive Margin RLHF via Preference over Preferences
by: Chittepu, Yaswanth, et al.
Published: (2025)

Is the Top Still Spinning? Evaluating Subjectivity in Narrative Understanding
by: Subbiah, Melanie, et al.
Published: (2025)

ChartMuseum: Testing Visual Reasoning Capabilities of Large Vision-Language Models
by: Tang, Liyan, et al.
Published: (2025)

VERA: Identifying and Leveraging Visual Evidence Retrieval Heads in Long-Context Understanding
by: Pei, Rongcan, et al.
Published: (2026)

VeriSoftBench: Repository-Scale Formal Verification Benchmarks for Lean
by: Xin, Yutong, et al.
Published: (2026)

RankAlign: A Ranking View of the Generator-Validator Gap in Large Language Models
by: Rodriguez, Juan Diego, et al.
Published: (2025)

Using Natural Language Explanations to Rescale Human Judgments
by: Wadhwa, Manya, et al.
Published: (2023)

A Long Way to Go: Investigating Length Correlations in RLHF
by: Singhal, Prasann, et al.
Published: (2023)

PropMEND: Hypernetworks for Knowledge Propagation in LLMs
by: Liu, Zeyu Leo, et al.
Published: (2025)

EndPrompt: Efficient Long-Context Extension via Terminal Anchoring
by: Tian, Han, et al.
Published: (2026)

D2PO: Discriminator-Guided DPO with Response Evaluation Models
by: Singhal, Prasann, et al.
Published: (2024)

SPARTA ALIGNMENT: Collectively Aligning Multiple Language Models through Combat
by: Jiang, Yuru, et al.
Published: (2025)

MuSR: Testing the Limits of Chain-of-thought with Multistep Soft Reasoning
by: Sprague, Zayne, et al.
Published: (2023)

Retrieval Head Mechanistically Explains Long-Context Factuality
by: Wu, Wenhao, et al.
Published: (2024)

Dehallucinating Parallel Context Extension for Retrieval-Augmented Generation
by: Ma, Zexiong, et al.
Published: (2024)

Understanding and Controlling Repetition Neurons and Induction Heads in In-Context Learning
by: Doan, Nhi Hoai, et al.
Published: (2025)

ACER: Automatic Language Model Context Extension via Retrieval
by: Gao, Luyu, et al.
Published: (2024)

Which questions should I answer? Salience Prediction of Inquisitive Questions
by: Wu, Yating, et al.
Published: (2024)

CREATE: Testing LLMs for Associative Creativity
by: Wadhwa, Manya, et al.
Published: (2026)

Understanding the RoPE Extensions of Long-Context LLMs: An Attention Perspective
by: Zhong, Meizhi, et al.
Published: (2024)

An Efficient Recipe for Long Context Extension via Middle-Focused Positional Encoding
by: Wu, Tong, et al.
Published: (2024)

Identifying Semantic Induction Heads to Understand In-Context Learning
by: Ren, Jie, et al.
Published: (2024)

ProofWala: A Framework for Multilingual Proof Data Synthesis and Theorem-Proving
by: Thakur, Amitayush, et al.
Published: (2025)

Grounding Long-Context Reasoning with Contextual Normalization for Retrieval-Augmented Generation
by: Chen, Jiamin, et al.
Published: (2025)