Image de couverture de livre

Summary of a Haystack: A Challenge to Long-Context LLMs and RAG Systems

Fuente: arXiv

Enregistré dans:

Détails bibliographiques
Auteurs principaux:	Laban, Philippe, Fabbri, Alexander R., Xiong, Caiming, Wu, Chien-Sheng
Format:	Preprint
Publié:	2024
Sujets:	Computation and Language
Accès en ligne:	Acceder al recurso
Tags:	Ajouter un tag Pas de tags, Soyez le premier à ajouter un tag!

Documents similaires

Are You Sure? Challenging LLMs Leads to Performance Drops in The FlipFlop Experiment
par: Laban, Philippe, et autres
Publié: (2023)

Do RAG Systems Cover What Matters? Evaluating and Optimizing Responses with Sub-Question Coverage
par: Xie, Kaige, et autres
Publié: (2024)

Embrace Divergence for Richer Insights: A Multi-document Summarization Benchmark and a Case Study on Summarizing Diverse Information from News Articles
par: Huang, Kung-Hsiang, et autres
Publié: (2023)

U-NIAH: Unified RAG and LLM Evaluation for Long Context Needle-In-A-Haystack
par: Gao, Yunfan, et autres
Publié: (2025)

SummExecEdit: A Factual Consistency Benchmark in Summarization with Executable Edits
par: Thorat, Onkar, et autres
Publié: (2024)

Shared Imagination: LLMs Hallucinate Alike
par: Zhou, Yilun, et autres
Publié: (2024)

SiReRAG: Indexing Similar and Related Information for Multihop Reasoning
par: Zhang, Nan, et autres
Publié: (2024)

Can AI writing be salvaged? Mitigating Idiosyncrasies and Improving Human-AI Alignment in the Writing Process through Edits
par: Chakrabarty, Tuhin, et autres
Publié: (2024)

Prompt Leakage effect and defense strategies for multi-turn LLM interactions
par: Agarwal, Divyansh, et autres
Publié: (2024)

AI-Slop to AI-Polish? Aligning Language Models through Edit-Based Writing Rewards and Test-time Computation
par: Chakrabarty, Tuhin, et autres
Publié: (2025)

Why Vision Language Models Struggle with Visual Arithmetic? Towards Enhanced Chart and Geometry Understanding
par: Huang, Kung-Hsiang, et autres
Publié: (2025)

UNIDOC-BENCH: A Unified Benchmark for Document-Centric Multimodal RAG
par: Peng, Xiangyu, et autres
Publié: (2025)

BABILong: Testing the Limits of LLMs with Long Context Reasoning-in-a-Haystack
par: Kuratov, Yuri, et autres
Publié: (2024)

Agentic Confidence Calibration
par: Zhang, Jiaxin, et autres
Publié: (2026)

Long-Context LLMs Meet RAG: Overcoming Challenges for Long Inputs in RAG
par: Jin, Bowen, et autres
Publié: (2024)

BingoGuard: LLM Content Moderation Tools with Risk Levels
par: Yin, Fan, et autres
Publié: (2025)

ReGenesis: LLMs can Grow into Reasoning Generalists via Self-Improvement
par: Peng, Xiangyu, et autres
Publié: (2024)

CRMArena: Understanding the Capacity of LLM Agents to Perform Professional CRM Tasks in Realistic Environments
par: Huang, Kung-Hsiang, et autres
Publié: (2024)

Unanswerability Evaluation for Retrieval Augmented Generation
par: Peng, Xiangyu, et autres
Publié: (2024)

Logic Haystacks: Probing LLMs Long-Context Logical Reasoning (Without Easily Identifiable Unrelated Padding)
par: Sileo, Damien
Publié: (2025)

Art or Artifice? Large Language Models and the False Promise of Creativity
par: Chakrabarty, Tuhin, et autres
Publié: (2023)

LLMs Corrupt Your Documents When You Delegate
par: Laban, Philippe, et autres
Publié: (2026)

Long Context vs. RAG for LLMs: An Evaluation and Revisits
par: Li, Xinze, et autres
Publié: (2024)

Convomem Benchmark: Why Your First 150 Conversations Don't Need RAG
par: Pakhomov, Egor, et autres
Publié: (2025)

MathHay: An Automated Benchmark for Long-Context Mathematical Reasoning in LLMs
par: Wang, Lei, et autres
Publié: (2024)

DeepTRACE: Auditing Deep Research AI Systems for Tracking Reliability Across Citations and Evidence
par: Venkit, Pranav Narayanan, et autres
Publié: (2025)

Haystack Engineering: Context Engineering for Heterogeneous and Agentic Long-Context Evaluation
par: Li, Mufei, et autres
Publié: (2025)

MiniCheck: Efficient Fact-Checking of LLMs on Grounding Documents
par: Tang, Liyan, et autres
Publié: (2024)

GUI-KV: Efficient GUI Agents via KV Cache with Spatio-Temporal Awareness
par: Huang, Kung-Hsiang, et autres
Publié: (2025)

Search Engines in an AI Era: The False Promise of Factual and Verifiable Source-Cited Responses
par: Venkit, Pranav Narayanan, et autres
Publié: (2024)

LLMs Get Lost In Multi-Turn Conversation
par: Laban, Philippe, et autres
Publié: (2025)

Michelangelo: Long Context Evaluations Beyond Haystacks via Latent Structure Queries
par: Vodrahalli, Kiran, et autres
Publié: (2024)

Less is More for Long Document Summary Evaluation by LLMs
par: Wu, Yunshu, et autres
Publié: (2023)

Turning Conversations into Workflows: A Framework to Extract and Evaluate Dialog Workflows for Service AI Agents
par: Choubey, Prafulla Kumar, et autres
Publié: (2025)

Multilingual Needle in a Haystack: Investigating Long-Context Behavior of Multilingual Large Language Models
par: Hengle, Amey, et autres
Publié: (2024)

Stress-Testing Long-Context Language Models with Lifelong ICL and Task Haystack
par: Xu, Xiaoyue, et autres
Publié: (2024)

Agentic Uncertainty Quantification
par: Zhang, Jiaxin, et autres
Publié: (2026)

Sequential-NIAH: A Needle-In-A-Haystack Benchmark for Extracting Sequential Needles from Long Contexts
par: Yu, Yifei, et autres
Publié: (2025)

GOLD PANNING: Strategic Context Shuffling for Needle-in-Haystack Reasoning
par: Byerly, Adam, et autres
Publié: (2025)

Jailbreaking in the Haystack
par: Shah, Rishi Rajesh, et autres
Publié: (2025)