Cover Image

Jackal: A Real-World Execution-Based Benchmark Evaluating Large Language Models on Text-to-JQL Tasks

Fuente: arXiv

Saved in:

Bibliographic Details
Main Authors:	Frank, Kevin, Gulati, Anmol, Lumer, Elias, Campagna, Sindy, Subbiah, Vamse Kumar
Format:	Preprint
Published:	2025
Subjects:	Computation and Language
Online Access:	Acceder al recurso
Tags:	Add Tag No Tags, Be the first to tag this record!

Similar Items

Agentic Jackal: Live Execution and Semantic Value Grounding for Text-to-JQL
by: Murali, Vishnu, et al.
Published: (2026)

Chronos: Temporal-Aware Conversational Agents with Structured Event Retrieval for Long-Term Memory
by: Sen, Sahil, et al.
Published: (2026)

Don't Break the Cache: An Evaluation of Prompt Caching for Long-Horizon Agentic Tasks
by: Lumer, Elias, et al.
Published: (2026)

Ask Early, Ask Late, Ask Right: When Does Clarification Timing Matter for Long-Horizon Agents?
by: Gulati, Anmol, et al.
Published: (2026)

Is Grep All You Need? How Agent Harnesses Reshape Agentic Search
by: Sen, Sahil, et al.
Published: (2026)

Agent-as-a-Graph: Knowledge Graph-Based Tool and Agent Retrieval for LLM Multi-Agent Systems
by: Nizar, Faheem, et al.
Published: (2025)

Tool-to-Agent Retrieval: Bridging Tools and Agents for Scalable LLM Multi-Agent Systems
by: Lumer, Elias, et al.
Published: (2025)

ScaleMCP: Dynamic and Auto-Synchronizing Model Context Protocol Tools for LLM Agents
by: Lumer, Elias, et al.
Published: (2025)

MemTool: Optimizing Short-Term Memory Management for Dynamic Tool Calling in LLM Agent Multi-Turn Conversations
by: Lumer, Elias, et al.
Published: (2025)

Comparison of Text-Based and Image-Based Retrieval in Multimodal Retrieval Augmented Generation Large Language Model Systems
by: Lumer, Elias, et al.
Published: (2025)

Cited but Not Verified: Parsing and Evaluating Source Attribution in LLM Deep Research Agents
by: Onweller, Hailey, et al.
Published: (2026)

Toolshed: Scale Tool-Equipped Agents with Advanced RAG-Tool Fusion and Tool Knowledge Bases
by: Lumer, Elias, et al.
Published: (2024)

Graph RAG-Tool Fusion
by: Lumer, Elias, et al.
Published: (2025)

Simulation Agent: A Framework for Integrating Simulation and Large Language Models for Enhanced Decision-Making
by: Kleiman, Jacob, et al.
Published: (2025)

Rethinking Retrieval: From Traditional Retrieval Augmented Generation to Agentic and Non-Vector Reasoning Systems in the Financial Domain for Large Language Models
by: Lumer, Elias, et al.
Published: (2025)

Evaluating Large Language Models for Real-World Engineering Tasks
by: Heesch, Rene, et al.
Published: (2025)

From Rows to Reasoning: A Retrieval-Augmented Multimodal Framework for Spreadsheet Understanding
by: Gulati, Anmol, et al.
Published: (2026)

Beyond Rows to Reasoning: Agentic Retrieval for Multimodal Spreadsheet Understanding and Editing
by: Gulati, Anmol, et al.
Published: (2026)

A Comprehensive Evaluation of Large Language Models on Benchmark Biomedical Text Processing Tasks
by: Jahan, Israt, et al.
Published: (2023)

RealFactBench: A Benchmark for Evaluating Large Language Models in Real-World Fact-Checking
by: Yang, Shuo, et al.
Published: (2025)

Reading Subtext: Evaluating Large Language Models on Short Story Summarization with Writers
by: Subbiah, Melanie, et al.
Published: (2024)

Text2World: Benchmarking Large Language Models for Symbolic World Model Generation
by: Hu, Mengkang, et al.
Published: (2025)

BRIDGE: Benchmarking Large Language Models for Understanding Real-world Clinical Practice Text
by: Wu, Jiageng, et al.
Published: (2025)

QuarkMedBench: A Real-World Scenario Driven Benchmark for Evaluating Large Language Models
by: Wu, Yao, et al.
Published: (2026)

Benchmarking Open-Source Large Language Models on Healthcare Text Classification Tasks
by: Guo, Yuting, et al.
Published: (2025)

Evaluating Large Language Models on Spatial Tasks: A Multi-Task Benchmarking Study
by: Xu, Liuchang, et al.
Published: (2024)

Evaluating the Effectiveness of Cost-Efficient Large Language Models in Benchmark Biomedical Tasks
by: Jahan, Israt, et al.
Published: (2025)

RWKU: Benchmarking Real-World Knowledge Unlearning for Large Language Models
by: Jin, Zhuoran, et al.
Published: (2024)

Evaluating Large Language Models in Crisis Detection: A Real-World Benchmark from Psychological Support Hotlines
by: Deng, Guifeng, et al.
Published: (2025)

MCP-SafetyBench: A Benchmark for Safety Evaluation of Large Language Models with Real-World MCP Servers
by: Zong, Xuanjun, et al.
Published: (2025)

MMRC: A Large-Scale Benchmark for Understanding Multimodal Large Language Model in Real-World Conversation
by: Xue, Haochen, et al.
Published: (2025)

PM-LLM-Benchmark: Evaluating Large Language Models on Process Mining Tasks
by: Berti, Alessandro, et al.
Published: (2024)

MCP-Universe: Benchmarking Large Language Models with Real-World Model Context Protocol Servers
by: Luo, Ziyang, et al.
Published: (2025)

Large Language Model Agent for Modular Task Execution in Drug Discovery
by: Ock, Janghoon, et al.
Published: (2025)

BAMBOO: A Comprehensive Benchmark for Evaluating Long Text Modeling Capacities of Large Language Models
by: Dong, Zican, et al.
Published: (2023)

TheAgentCompany: Benchmarking LLM Agents on Consequential Real World Tasks
by: Xu, Frank F., et al.
Published: (2024)

TaskBench: Benchmarking Large Language Models for Task Automation
by: Shen, Yongliang, et al.
Published: (2023)

Large Language Model Benchmarks in Medical Tasks
by: Yan, Lawrence K. Q., et al.
Published: (2024)

QuantCode-Bench: A Benchmark for Evaluating the Ability of Large Language Models to Generate Executable Algorithmic Trading Strategies
by: Khoroshilov, Alexey, et al.
Published: (2026)

ViTaB-A: Evaluating Multimodal Large Language Models on Visual Table Attribution
by: Alqurnawi, Yahia, et al.
Published: (2026)