Hierarchical Attention Generates Better Proofs
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2025
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866910920404369408 |
|---|---|
| author | Chen, Jianlong Li, Chao Yuan, Yang Yao, Andrew C |
| author_facet | Chen, Jianlong Li, Chao Yuan, Yang Yao, Andrew C |
| contents | Large language models (LLMs) have shown promise in formal theorem proving, but their token-level processing often fails to capture the inherent hierarchical nature of mathematical proofs. We introduce \textbf{Hierarchical Attention}, a regularization method that aligns LLMs' attention mechanisms with mathematical reasoning structures. Our approach establishes a five-level hierarchy from foundational elements to high-level concepts, ensuring structured information flow in proof generation. Experiments demonstrate that our method improves proof success rates by 2.05\% on miniF2F and 1.69\% on ProofNet while reducing proof complexity by 23.81\% and 16.50\% respectively. The code is available at https://github.com/Car-pe/HAGBP. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2504_19188 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Hierarchical Attention Generates Better Proofs Chen, Jianlong Li, Chao Yuan, Yang Yao, Andrew C Machine Learning Artificial Intelligence Computation and Language Logic in Computer Science Large language models (LLMs) have shown promise in formal theorem proving, but their token-level processing often fails to capture the inherent hierarchical nature of mathematical proofs. We introduce \textbf{Hierarchical Attention}, a regularization method that aligns LLMs' attention mechanisms with mathematical reasoning structures. Our approach establishes a five-level hierarchy from foundational elements to high-level concepts, ensuring structured information flow in proof generation. Experiments demonstrate that our method improves proof success rates by 2.05\% on miniF2F and 1.69\% on ProofNet while reducing proof complexity by 23.81\% and 16.50\% respectively. The code is available at https://github.com/Car-pe/HAGBP. |
| title | Hierarchical Attention Generates Better Proofs |
| topic | Machine Learning Artificial Intelligence Computation and Language Logic in Computer Science |
| url | https://arxiv.org/abs/2504.19188 |