SorryDB: Can AI Provers Complete Real-World Lean Theorems?
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2026
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866912939264442368 |
|---|---|
| author | Letson, Austin Sarra, Leopoldo Poiroux, Auguste Dressler, Oliver Lezeau, Paul Aranha, Dhyan Pu, Frederick Hill, Aaron Hidalgo, Miguel Corredera Berman, Julian Tsoukalas, George Taelman, Lenny |
| author_facet | Letson, Austin Sarra, Leopoldo Poiroux, Auguste Dressler, Oliver Lezeau, Paul Aranha, Dhyan Pu, Frederick Hill, Aaron Hidalgo, Miguel Corredera Berman, Julian Tsoukalas, George Taelman, Lenny |
| contents | We present SorryDB, a dynamically-updating benchmark of open Lean tasks drawn from 78 real world formalization projects on GitHub. Unlike existing static benchmarks, often composed of competition problems, hillclimbing the SorryDB benchmark will yield tools that are aligned to the community needs, more usable by mathematicians, and more capable of understanding complex dependencies. Moreover, by providing a continuously updated stream of tasks, SorryDB mitigates test-set contamination and offers a robust metric for an agent's ability to contribute to novel formal mathematics projects. We evaluate a collection of approaches, including generalist large language models, agentic approaches, and specialized symbolic provers, over a selected snapshot of 1000 tasks from SorryDB. We show that current approaches are complementary: even though an agentic approach based on Gemini Flash is the most performant, it is not strictly better than other off-the-shelf large-language models, specialized provers, or even a curated list of Lean tactics. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2603_02668 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | SorryDB: Can AI Provers Complete Real-World Lean Theorems? Letson, Austin Sarra, Leopoldo Poiroux, Auguste Dressler, Oliver Lezeau, Paul Aranha, Dhyan Pu, Frederick Hill, Aaron Hidalgo, Miguel Corredera Berman, Julian Tsoukalas, George Taelman, Lenny Artificial Intelligence Machine Learning We present SorryDB, a dynamically-updating benchmark of open Lean tasks drawn from 78 real world formalization projects on GitHub. Unlike existing static benchmarks, often composed of competition problems, hillclimbing the SorryDB benchmark will yield tools that are aligned to the community needs, more usable by mathematicians, and more capable of understanding complex dependencies. Moreover, by providing a continuously updated stream of tasks, SorryDB mitigates test-set contamination and offers a robust metric for an agent's ability to contribute to novel formal mathematics projects. We evaluate a collection of approaches, including generalist large language models, agentic approaches, and specialized symbolic provers, over a selected snapshot of 1000 tasks from SorryDB. We show that current approaches are complementary: even though an agentic approach based on Gemini Flash is the most performant, it is not strictly better than other off-the-shelf large-language models, specialized provers, or even a curated list of Lean tactics. |
| title | SorryDB: Can AI Provers Complete Real-World Lean Theorems? |
| topic | Artificial Intelligence Machine Learning |
| url | https://arxiv.org/abs/2603.02668 |