Cover Image

Self-Consistency Falls Short! The Adverse Effects of Positional Bias on Long-Context Problems

Fuente: arXiv

Saved in:

Bibliographic Details
Main Authors:	Byerly, Adam, Khashabi, Daniel
Format:	Preprint
Published:	2024
Subjects:	Computation and Language
Online Access:	Acceder al recurso
Tags:	Add Tag No Tags, Be the first to tag this record!

Similar Items

GOLD PANNING: Strategic Context Shuffling for Needle-in-Haystack Reasoning
by: Byerly, Adam, et al.
Published: (2025)

Insights into LLM Long-Context Failures: When Transformers Know but Don't Tell
by: Lu, Taiming, et al.
Published: (2024)

Long Context Alignment with Short Instructions and Synthesized Positions
by: Wu, Wenhao, et al.
Published: (2024)

Layer-wise Positional Bias in Short-Context Language Modeling
by: Rahimi, Maryam, et al.
Published: (2026)

LongBEL: Long-Context and Document-Consistent Biomedical Entity Linking
by: Remaki, Adam, et al.
Published: (2026)

Short Data, Long Context: Distilling Positional Knowledge in Transformers
by: Huber, Patrick, et al.
Published: (2026)

Do pretrained Transformers Learn In-Context by Gradient Descent?
by: Shen, Lingfeng, et al.
Published: (2023)

Why Does the Effective Context Length of LLMs Fall Short?
by: An, Chenxin, et al.
Published: (2024)

Challenging the Evaluator: LLM Sycophancy Under User Rebuttal
by: Kim, Sungwon, et al.
Published: (2025)

SIMPLEMIX: Frustratingly Simple Mixing of Off- and On-policy Data in Language Model Preference Learning
by: Li, Tianjian, et al.
Published: (2025)

Position Bias Mitigates Position Bias:Mitigate Position Bias Through Inter-Position Knowledge Distillation
by: Wang, Yifei, et al.
Published: (2025)

ICL CIPHERS: Quantifying "Learning" in In-Context Learning via Substitution Ciphers
by: Fang, Zhouxiang, et al.
Published: (2025)

On Positional Bias of Faithfulness for Long-form Summarization
by: Wan, David, et al.
Published: (2024)

Found in the Middle: Calibrating Positional Attention Bias Improves Long Context Utilization
by: Hsieh, Cheng-Yu, et al.
Published: (2024)

Latent Self-Consistency for Reliable Majority-Set Selection in Short- and Long-Answer Reasoning
by: Oh, Jungsuk, et al.
Published: (2025)

DiffNorm: Self-Supervised Normalization for Non-autoregressive Speech-to-speech Translation
by: Tan, Weiting, et al.
Published: (2024)

Beyond Position Bias: Shifting Context Compression from Position-Driven to Semantic-Driven
by: Tang, Jiwei, et al.
Published: (2026)

LongPO: Long Context Self-Evolution of Large Language Models through Short-to-Long Preference Optimization
by: Chen, Guanzheng, et al.
Published: (2025)

Evaluating the Evaluators: Are readability metrics good measures of readability?
by: Cachola, Isabel, et al.
Published: (2025)

Wavelet-based Positional Representation for Long Context
by: Oka, Yui, et al.
Published: (2025)

Tur[k]ingBench: A Challenge Benchmark for Web Agents
by: Xu, Kevin, et al.
Published: (2024)

Long-Short Alignment for Effective Long-Context Modeling in LLMs
by: Du, Tianqi, et al.
Published: (2025)

SEGMENT+: Long Text Processing with Short-Context Language Models
by: Shi, Wei, et al.
Published: (2024)

Principled Context Engineering for RAG: Statistical Guarantees via Conformal Prediction
by: Chakraborty, Debashish, et al.
Published: (2025)

Atomic Self-Consistency for Better Long Form Generations
by: Thirukovalluru, Raghuveer, et al.
Published: (2024)

Trust Functions: Near-Lossless Weak-to-Strong Generalization by Learning When to Trust the Weak Teacher
by: Uzunoglu, Arda, et al.
Published: (2026)

The Flaw of Averages: Quantifying Uniformity of Performance on Benchmarks
by: Uzunoglu, Arda, et al.
Published: (2025)

Positional Bias in Long-Document Ranking: Impact, Assessment, and Mitigation
by: Boytsov, Leonid, et al.
Published: (2022)

Layer-Specific Scaling of Positional Encodings for Superior Long-Context Modeling
by: Wang, Zhenghua, et al.
Published: (2025)

Training-Inference Consistent Segmented Execution for Long-Context LLMs
by: Shang, Xianpeng, et al.
Published: (2026)

Where to show Demos in Your Prompt: A Positional Bias of In-Context Learning
by: Cobbina, Kwesi, et al.
Published: (2025)

Generalizing From Short to Long: Effective Data Synthesis for Long-Context Instruction Tuning
by: Zhu, Wenhao, et al.
Published: (2025)

Tokenization Falling Short: On Subword Robustness in Large Language Models
by: Chai, Yekun, et al.
Published: (2024)

Shuffle the Context: RoPE-Perturbed Self-Distillation for Long-Context Adaptation
by: Li, Zichong, et al.
Published: (2026)

IA2: Alignment with ICL Activations Improves Supervised Fine-Tuning
by: Mishra, Aayush, et al.
Published: (2025)

WorldAPIs: The World Is Worth How Many APIs? A Thought Experiment
by: Ou, Jiefu, et al.
Published: (2024)

Are Finer Citations Always Better? Rethinking Granularity for Attributed Generation
by: Wang, Hexuan, et al.
Published: (2026)

An Efficient Recipe for Long Context Extension via Middle-Focused Positional Encoding
by: Wu, Tong, et al.
Published: (2024)

Mitigating Posterior Salience Attenuation in Long-Context LLMs with Positional Contrastive Decoding
by: Xiao, Zikai, et al.
Published: (2025)

Beyond Real: Imaginary Extension of Rotary Position Embeddings for Long-Context LLMs
by: Liu, Xiaoran, et al.
Published: (2025)