CryptoX : Compositional Reasoning Evaluation of Large Language Models
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2025
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866929755049164800 |
|---|---|
| author | Shi, Jiajun Wei, Chaoren Yang, Liqun Wang, Zekun Moore Yang, Chenghao Zhang, Ge Huang, Stephen Peng, Tao Yang, Jian Wen, Zhoufutu |
| author_facet | Shi, Jiajun Wei, Chaoren Yang, Liqun Wang, Zekun Moore Yang, Chenghao Zhang, Ge Huang, Stephen Peng, Tao Yang, Jian Wen, Zhoufutu |
| contents | The compositional reasoning capacity has long been regarded as critical to the generalization and intelligence emergence of large language models LLMs. However, despite numerous reasoning-related benchmarks, the compositional reasoning capacity of LLMs is rarely studied or quantified in the existing benchmarks. In this paper, we introduce CryptoX, an evaluation framework that, for the first time, combines existing benchmarks and cryptographic, to quantify the compositional reasoning capacity of LLMs. Building upon CryptoX, we construct CryptoBench, which integrates these principles into several benchmarks for systematic evaluation. We conduct detailed experiments on widely used open-source and closed-source LLMs using CryptoBench, revealing a huge gap between open-source and closed-source LLMs. We further conduct thorough mechanical interpretability experiments to reveal the inner mechanism of LLMs' compositional reasoning, involving subproblem decomposition, subproblem inference, and summarizing subproblem conclusions. Through analysis based on CryptoBench, we highlight the value of independently studying compositional reasoning and emphasize the need to enhance the compositional reasoning capabilities of LLMs. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2502_07813 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | CryptoX : Compositional Reasoning Evaluation of Large Language Models Shi, Jiajun Wei, Chaoren Yang, Liqun Wang, Zekun Moore Yang, Chenghao Zhang, Ge Huang, Stephen Peng, Tao Yang, Jian Wen, Zhoufutu Cryptography and Security Artificial Intelligence The compositional reasoning capacity has long been regarded as critical to the generalization and intelligence emergence of large language models LLMs. However, despite numerous reasoning-related benchmarks, the compositional reasoning capacity of LLMs is rarely studied or quantified in the existing benchmarks. In this paper, we introduce CryptoX, an evaluation framework that, for the first time, combines existing benchmarks and cryptographic, to quantify the compositional reasoning capacity of LLMs. Building upon CryptoX, we construct CryptoBench, which integrates these principles into several benchmarks for systematic evaluation. We conduct detailed experiments on widely used open-source and closed-source LLMs using CryptoBench, revealing a huge gap between open-source and closed-source LLMs. We further conduct thorough mechanical interpretability experiments to reveal the inner mechanism of LLMs' compositional reasoning, involving subproblem decomposition, subproblem inference, and summarizing subproblem conclusions. Through analysis based on CryptoBench, we highlight the value of independently studying compositional reasoning and emphasize the need to enhance the compositional reasoning capabilities of LLMs. |
| title | CryptoX : Compositional Reasoning Evaluation of Large Language Models |
| topic | Cryptography and Security Artificial Intelligence |
| url | https://arxiv.org/abs/2502.07813 |