Saved in:
| Main Authors: | , , |
|---|---|
| Format: | Preprint |
| Published: |
2026
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2604.27340 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866917450200645632 |
|---|---|
| author | Xu, Ziyao Wang, Cong Wang, Houfeng |
| author_facet | Xu, Ziyao Wang, Cong Wang, Houfeng |
| contents | Compositional generalization tests are often used to estimate the compositionality of LLMs. However, such tests have the following limitations: (1) they only focus on the output results without considering LLMs' understanding of sample compositionality, resulting in explainability defects; (2) they rely on dataset partition to form the test set with combinations unseen in the training set, suffering from combination leakage issues. In this work, we propose a novel rule-generation perspective for compositionality estimation for LLMs. It requires LLMs to generate a program as rules for dataset mapping and provides estimates of the compositionality of LLMs using complexity-based theory. The perspective addresses the limitations of compositional generalization tests and provides a new way to analyze the compositionality characterization of LLMs. We conduct experiments and analysis of existing advanced LLMs based on this perspective on a string-to-grid task, and find various compositionality characterizations and compositionality deficiencies exhibited by LLMs. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2604_27340 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | Investigating More Explainable and Partition-Free Compositionality Estimation for LLMs: A Rule-Generation Perspective Xu, Ziyao Wang, Cong Wang, Houfeng Artificial Intelligence Compositional generalization tests are often used to estimate the compositionality of LLMs. However, such tests have the following limitations: (1) they only focus on the output results without considering LLMs' understanding of sample compositionality, resulting in explainability defects; (2) they rely on dataset partition to form the test set with combinations unseen in the training set, suffering from combination leakage issues. In this work, we propose a novel rule-generation perspective for compositionality estimation for LLMs. It requires LLMs to generate a program as rules for dataset mapping and provides estimates of the compositionality of LLMs using complexity-based theory. The perspective addresses the limitations of compositional generalization tests and provides a new way to analyze the compositionality characterization of LLMs. We conduct experiments and analysis of existing advanced LLMs based on this perspective on a string-to-grid task, and find various compositionality characterizations and compositionality deficiencies exhibited by LLMs. |
| title | Investigating More Explainable and Partition-Free Compositionality Estimation for LLMs: A Rule-Generation Perspective |
| topic | Artificial Intelligence |
| url | https://arxiv.org/abs/2604.27340 |