BadScientist: Can a Research Agent Write Convincing but Unsound Papers that Fool LLM Reviewers?
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866912662495952896 |
|---|---|
| author | Jiang, Fengqing Feng, Yichen Li, Yuetai Niu, Luyao Alomair, Basel Poovendran, Radha |
| author_facet | Jiang, Fengqing Feng, Yichen Li, Yuetai Niu, Luyao Alomair, Basel Poovendran, Radha |
| contents | The convergence of LLM-powered research assistants and AI-based peer review systems creates a critical vulnerability: fully automated publication loops where AI-generated research is evaluated by AI reviewers without human oversight. We investigate this through \textbf{BadScientist}, a framework that evaluates whether fabrication-oriented paper generation agents can deceive multi-model LLM review systems. Our generator employs presentation-manipulation strategies requiring no real experiments. We develop a rigorous evaluation framework with formal error guarantees (concentration bounds and calibration analysis), calibrated on real data. Our results reveal systematic vulnerabilities: fabricated papers achieve acceptance rates up to . Critically, we identify \textit{concern-acceptance conflict} -- reviewers frequently flag integrity issues yet assign acceptance-level scores. Our mitigation strategies show only marginal improvements, with detection accuracy barely exceeding random chance. Despite provably sound aggregation mathematics, integrity checking systematically fails, exposing fundamental limitations in current AI-driven review systems and underscoring the urgent need for defense-in-depth safeguards in scientific publishing. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2510_18003 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | BadScientist: Can a Research Agent Write Convincing but Unsound Papers that Fool LLM Reviewers? Jiang, Fengqing Feng, Yichen Li, Yuetai Niu, Luyao Alomair, Basel Poovendran, Radha Cryptography and Security Artificial Intelligence Computers and Society The convergence of LLM-powered research assistants and AI-based peer review systems creates a critical vulnerability: fully automated publication loops where AI-generated research is evaluated by AI reviewers without human oversight. We investigate this through \textbf{BadScientist}, a framework that evaluates whether fabrication-oriented paper generation agents can deceive multi-model LLM review systems. Our generator employs presentation-manipulation strategies requiring no real experiments. We develop a rigorous evaluation framework with formal error guarantees (concentration bounds and calibration analysis), calibrated on real data. Our results reveal systematic vulnerabilities: fabricated papers achieve acceptance rates up to . Critically, we identify \textit{concern-acceptance conflict} -- reviewers frequently flag integrity issues yet assign acceptance-level scores. Our mitigation strategies show only marginal improvements, with detection accuracy barely exceeding random chance. Despite provably sound aggregation mathematics, integrity checking systematically fails, exposing fundamental limitations in current AI-driven review systems and underscoring the urgent need for defense-in-depth safeguards in scientific publishing. |
| title | BadScientist: Can a Research Agent Write Convincing but Unsound Papers that Fool LLM Reviewers? |
| topic | Cryptography and Security Artificial Intelligence Computers and Society |
| url | https://arxiv.org/abs/2510.18003 |