Cover Image

Is this Idea Novel? An Automated Benchmark for Judgment of Research Ideas

Fuente: arXiv

Saved in:

Bibliographic Details
Main Authors:	Schopf, Tim, Färber, Michael
Format:	Preprint
Published:	2026
Subjects:	Computation and Language Artificial Intelligence I.2.7
Online Access:	Acceder al recurso
Tags:	Add Tag No Tags, Be the first to tag this record!

Similar Items

Low-Resource Court Judgment Summarization for Common Law Systems
by: Liu, Shuaiqi, et al.
Published: (2024)

EvoIdeator: Evolving Scientific Ideas through Checklist-Grounded Reinforcement Learning
by: Sauter, Andreas, et al.
Published: (2026)

DeEscalWild: A Real-World Benchmark for Automated De-Escalation Training with SLMs
by: Hasan, Md Hasebul, et al.
Published: (2026)

RomanLens: The Role Of Latent Romanization In Multilinguality In LLMs
by: Saji, Alan, et al.
Published: (2025)

Text-Based Approaches to Item Difficulty Modeling in Large-Scale Assessments: A Systematic Review
by: Peters, Sydney, et al.
Published: (2025)

Efficient Few-shot Learning for Multi-label Classification of Scientific Documents with Many Classes
by: Schopf, Tim, et al.
Published: (2024)

Separating Constraint Compliance from Semantic Accuracy: A Novel Benchmark for Evaluating Instruction-Following Under Compression
by: Baxi, Rahul
Published: (2025)

Automated MCQA Benchmarking at Scale: Evaluating Reasoning Traces as Retrieval Sources for Domain Adaptation of Small Language Models
by: Gokdemir, Ozan, et al.
Published: (2025)

Graph Your Way to Inspiration: Integrating Co-Author Graphs with Retrieval-Augmented Generation for Large Language Model Based Scientific Idea Generation
by: Xie, Pengzhen, et al.
Published: (2025)

A Hierarchical Error Framework for Reliable Automated Coding in Communication Research: Applications to Health and Political Communication
by: Zhao, Zhilong, et al.
Published: (2025)

AMEL: Accumulated Message Effects on LLM Judgments
by: Temkit, Sid-Ali
Published: (2026)

Cultural Benchmarking of LLMs in Standard and Dialectal Arabic Dialogues
by: Kautsar, Muhammad Dehan Al, et al.
Published: (2026)

UA-Legal-Bench: A Benchmark for Evaluating Large Language Models on Ukrainian Legal Reasoning
by: Ovcharov, Volodymyr
Published: (2026)

PARAPHRASUS : A Comprehensive Benchmark for Evaluating Paraphrase Detection Models
by: Michail, Andrianos, et al.
Published: (2024)

CausalGym: Benchmarking causal interpretability methods on linguistic tasks
by: Arora, Aryaman, et al.
Published: (2024)

EQ-Bench: An Emotional Intelligence Benchmark for Large Language Models
by: Paech, Samuel J.
Published: (2023)

SUBLLM: A Novel Efficient Architecture with Token Sequence Subsampling for LLM
by: Wang, Quandong, et al.
Published: (2024)

PrivacyBench: A Conversational Benchmark for Evaluating Privacy in Personalized AI
by: Mukhopadhyay, Srija, et al.
Published: (2025)

Benchmarking the Performance of Pre-trained LLMs across Urdu NLP Tasks
by: Tahir, Munief Hassan, et al.
Published: (2024)

AI Appeals Processor: A Deep Learning Approach to Automated Classification of Citizen Appeals in Government Services
by: Beskorovainyi, Vladimir
Published: (2026)

PSST: A Benchmark for Evaluation-driven Text Public-Speaking Style Transfer
by: Sun, Huashan, et al.
Published: (2023)

LeCoDe: A Benchmark Dataset for Interactive Legal Consultation Dialogue Evaluation
by: Yuan, Weikang, et al.
Published: (2025)

EduAdapt: A Question Answer Benchmark Dataset for Evaluating Grade-Level Adaptability in LLMs
by: Naeem, Numaan, et al.
Published: (2025)

Cetvel: A Unified Benchmark for Evaluating Language Understanding, Generation and Cultural Capacity of LLMs for Turkish
by: Er, Yakup Abrek, et al.
Published: (2025)

No Dataset Needed for Downstream Knowledge Benchmarking: Response Dispersion Inversely Correlates with Accuracy on Domain-specific QA
by: Simione II, Robert L
Published: (2024)

FIN-bench-v2: A Unified and Robust Benchmark Suite for Evaluating Finnish Large Language Models
by: Kytöniemi, Joona, et al.
Published: (2025)

Evaluating the Efficacy of Hybrid Deep Learning Models in Distinguishing AI-Generated Text
by: Oketunji, Abiodun Finbarrs
Published: (2023)

UNO-Bench: A Unified Benchmark for Exploring the Compositional Law Between Uni-modal and Omni-modal in Omni Models
by: Chen, Chen, et al.
Published: (2025)

Towards Safer Chatbots: Automated Policy Compliance Evaluation of Custom GPTs
by: Rodriguez, David, et al.
Published: (2025)

RMGAP: Benchmarking the Generalization of Reward Models across Diverse Preferences
by: Zhou, Yangyang, et al.
Published: (2026)

Exploring Graph Representations of Logical Forms for Language Modeling
by: Sullivan, Michael
Published: (2025)

Can AI Read Between The Lines? Benchmarking LLMs On Financial Nuance
by: Kubica, Dominick, et al.
Published: (2025)

ChatGPT, Let us Chat Sign Language: Experiments, Architectural Elements, Challenges and Research Directions
by: Shahin, Nada, et al.
Published: (2024)

KAConvText: Novel Approach to Burmese Sentence Classification using Kolmogorov-Arnold Convolution
by: Thu, Ye Kyaw, et al.
Published: (2025)

Latent Planning Emerges with Scale
by: Hanna, Michael, et al.
Published: (2026)

UrduBench: An Urdu Reasoning Benchmark using Contextually Ensembled Translations with Human-in-the-Loop
by: Shafique, Muhammad Ali, et al.
Published: (2026)

From Benchmarking to Reasoning: A Dual-Aspect, Large-Scale Evaluation of LLMs on Vietnamese Legal Text
by: Le, Van-Truong
Published: (2026)

Argumentatively Coherent Judgmental Forecasting
by: Gorur, Deniz, et al.
Published: (2025)

Whether, Not Which: Mechanistic Interpretability Reveals Dissociable Affect Reception and Emotion Categorization in LLMs
by: Keeman, Michael
Published: (2026)

Large Language Model (LLM) Bias Index -- LLMBI
by: Oketunji, Abiodun Finbarrs, et al.
Published: (2023)