Enregistré dans:
| Auteurs principaux: | , , , , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2025
|
| Sujets: | |
| Accès en ligne: | https://arxiv.org/abs/2505.00001 |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866908346867515392 |
|---|---|
| author | Baek, Shaun Esua-Mensah, Shaun Tsui, Cyrus Vigneswaralingam, Sejan Alali, Abdullah Lu, Michael Sharma, Vasu O'Brien, Sean Zhu, Kevin |
| author_facet | Baek, Shaun Esua-Mensah, Shaun Tsui, Cyrus Vigneswaralingam, Sejan Alali, Abdullah Lu, Michael Sharma, Vasu O'Brien, Sean Zhu, Kevin |
| contents | Large Language Models (LLMs) are primarily trained on high-resource natural languages, limiting their effectiveness in low-resource settings and in tasks requiring deep logical reasoning. This research introduces Rosetta-PL, a benchmark designed to evaluate LLMs' logical reasoning and generalization capabilities in a controlled environment. We construct Rosetta-PL by translating a dataset of logical propositions from Lean into a custom logical language, which is then used to fine-tune an LLM (e.g., GPT-4o). Our experiments analyze the impact of the size of the dataset and the translation methodology on the performance of the model. Our results indicate that preserving logical relationships in the translation process significantly boosts precision, with accuracy plateauing beyond roughly 20,000 training samples. These insights provide valuable guidelines for optimizing LLM training in formal reasoning tasks and improving performance in various low-resource language applications. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2505_00001 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Rosetta-PL: Propositional Logic as a Benchmark for Large Language Model Reasoning Baek, Shaun Esua-Mensah, Shaun Tsui, Cyrus Vigneswaralingam, Sejan Alali, Abdullah Lu, Michael Sharma, Vasu O'Brien, Sean Zhu, Kevin Computation and Language Large Language Models (LLMs) are primarily trained on high-resource natural languages, limiting their effectiveness in low-resource settings and in tasks requiring deep logical reasoning. This research introduces Rosetta-PL, a benchmark designed to evaluate LLMs' logical reasoning and generalization capabilities in a controlled environment. We construct Rosetta-PL by translating a dataset of logical propositions from Lean into a custom logical language, which is then used to fine-tune an LLM (e.g., GPT-4o). Our experiments analyze the impact of the size of the dataset and the translation methodology on the performance of the model. Our results indicate that preserving logical relationships in the translation process significantly boosts precision, with accuracy plateauing beyond roughly 20,000 training samples. These insights provide valuable guidelines for optimizing LLM training in formal reasoning tasks and improving performance in various low-resource language applications. |
| title | Rosetta-PL: Propositional Logic as a Benchmark for Large Language Model Reasoning |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2505.00001 |