Cover Image

Automatic Reviewers Fail to Detect Faulty Reasoning in Research Papers: A New Counterfactual Evaluation Framework

Fuente: arXiv

Saved in:

Bibliographic Details
Main Authors:	Dycke, Nils, Gurevych, Iryna
Format:	Preprint
Published:	2025
Subjects:	Computation and Language
Online Access:	Acceder al recurso
Tags:	Add Tag No Tags, Be the first to tag this record!

Similar Items

STRICTA: Structured Reasoning in Critical Text Assessment for Peer Review and Beyond
by: Dycke, Nils, et al.
Published: (2024)

Decision-Making with Deliberation: Meta-reviewing as a Document-grounded Dialogue
by: Purkayastha, Sukannya, et al.
Published: (2025)

SciCoQA: Quality Assurance for Scientific Paper--Code Alignment
by: Baumgärtner, Tim, et al.
Published: (2026)

CATfOOD: Counterfactual Augmented Training for Improving Out-of-Domain Performance and Calibration
by: Sachdeva, Rachneet, et al.
Published: (2023)

The Good, the Bad and the Constructive: Automatically Measuring Peer Review's Utility for Authors
by: Sadallah, Abdelrahman, et al.
Published: (2025)

DAPR: A Benchmark on Document-Aware Passage Retrieval
by: Wang, Kexin, et al.
Published: (2023)

Tools Fail: Detecting Silent Errors in Faulty Tools
by: Sun, Jimin, et al.
Published: (2024)

Identifying Aspects in Peer Reviews
by: Lu, Sheng, et al.
Published: (2025)

Commitment Checklist: Auditing Author Commitments in Peer Review
by: Chen, Chung-Chi, et al.
Published: (2026)

Re3: A Holistic Framework and Dataset for Modeling Collaborative Document Revision
by: Ruan, Qian, et al.
Published: (2024)

DARA: Decomposition-Alignment-Reasoning Autonomous Language Agent for Question Answering over Knowledge Graphs
by: Fang, Haishuo, et al.
Published: (2024)

Reward Modeling for Scientific Writing Evaluation
by: Şahinuç, Furkan, et al.
Published: (2026)

The Nature of NLP: Analyzing Contributions in NLP Papers
by: Pramanick, Aniket, et al.
Published: (2024)

SPARE: Single-Pass Annotation with Reference-Guided Evaluation for Automatic Process Supervision and Reward Modelling
by: Rizvi, Md Imbesat Hassan, et al.
Published: (2025)

Expert Preference-based Evaluation of Automated Related Work Generation
by: Şahinuç, Furkan, et al.
Published: (2025)

GRITHopper: Decomposition-Free Multi-Hop Dense Retrieval
by: Erker, Justus-Jonas, et al.
Published: (2025)

Citation Failure: Definition, Analysis and Efficient Mitigation
by: Buchmann, Jan, et al.
Published: (2025)

Like a Good Nearest Neighbor: Practical Content Moderation and Text Classification
by: Bates, Luke, et al.
Published: (2023)

Co-FactChecker: A Framework for Human-AI Collaborative Claim Verification Using Large Reasoning Models
by: Sahnan, Dhruv, et al.
Published: (2026)

Socratic Reasoning Improves Positive Text Rewriting
by: Goel, Anmol, et al.
Published: (2024)

Is Peer Review Really in Decline? Analyzing Review Quality across Venues and Time
by: Kuznetsov, Ilia, et al.
Published: (2026)

Preemptive Detection and Correction of Misaligned Actions in LLM Agents
by: Fang, Haishuo, et al.
Published: (2024)

PeerQA: A Scientific Question Answering Dataset from Peer Reviews
by: Baumgärtner, Tim, et al.
Published: (2025)

NewsRECON: News article REtrieval for image CONtextualization
by: Tonglet, Jonathan, et al.
Published: (2026)

LazyReview A Dataset for Uncovering Lazy Thinking in NLP Peer Reviews
by: Purkayastha, Sukannya, et al.
Published: (2025)

Self-Rationalization in the Wild: A Large Scale Out-of-Distribution Evaluation on NLI-related tasks
by: Yang, Jing, et al.
Published: (2025)

Reviewing the Reviewer: Elevating Peer Review Quality through LLM-Guided Feedback
by: Purkayastha, Sukannya, et al.
Published: (2026)

Are Multilingual LLMs Culturally-Diverse Reasoners? An Investigation into Multicultural Proverbs and Sayings
by: Liu, Chen Cecilia, et al.
Published: (2023)

Are Large Language Models Good Classifiers? A Study on Edit Intent Classification in Scientific Document Revisions
by: Ruan, Qian, et al.
Published: (2024)

How to Handle Different Types of Out-of-Distribution Scenarios in Computational Argumentation? A Comprehensive and Fine-Grained Field Study
by: Waldis, Andreas, et al.
Published: (2023)

Exposía: Teaching and Assessment of Academic Writing Skills for Research Project Proposals and Peer Feedback
by: Zyska, Dennis, et al.
Published: (2026)

Code Prompting Elicits Conditional Reasoning Abilities in Text+Code LLMs
by: Puerto, Haritz, et al.
Published: (2024)

Token Weighting for Long-Range Language Modeling
by: Helm, Falko, et al.
Published: (2025)

COVE: COntext and VEracity prediction for out-of-context images
by: Tonglet, Jonathan, et al.
Published: (2025)

Turning Logic Against Itself : Probing Model Defenses Through Contrastive Questions
by: Sachdeva, Rachneet, et al.
Published: (2025)

M4FC: a Multimodal, Multilingual, Multicultural, Multitask Real-World Fact-Checking Dataset
by: Geng, Jiahui, et al.
Published: (2025)

Robust Utility-Preserving Text Anonymization Based on Large Language Models
by: Yang, Tianyu, et al.
Published: (2024)

Dive into the Chasm: Probing the Gap between In- and Cross-Topic Generalization
by: Waldis, Andreas, et al.
Published: (2024)

Overview of PerpectiveArg2024: The First Shared Task on Perspective Argument Retrieval
by: Falk, Neele, et al.
Published: (2024)

LLM Roleplay: Simulating Human-Chatbot Interaction
by: Tamoyan, Hovhannes, et al.
Published: (2024)