Saved in:
| Main Authors: | , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2509.19849 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866909804077776896 |
|---|---|
| author | Voigt, Henrik Kahlmeyer, Paul Lawonn, Kai Habeck, Michael Giesen, Joachim |
| author_facet | Voigt, Henrik Kahlmeyer, Paul Lawonn, Kai Habeck, Michael Giesen, Joachim |
| contents | Symbolic regression algorithms search a space of mathematical expressions for formulas that explain given data. Transformer-based models have emerged as a promising, scalable approach shifting the expensive combinatorial search to a large-scale pre-training phase. However, the success of these models is critically dependent on their pre-training data. Their ability to generalize to problems outside of this pre-training distribution remains largely unexplored. In this work, we conduct a systematic empirical study to evaluate the generalization capabilities of pre-trained, transformer-based symbolic regression. We rigorously test performance both within the pre-training distribution and on a series of out-of-distribution challenges for several state of the art approaches. Our findings reveal a significant dichotomy: while pre-trained models perform well in-distribution, the performance consistently degrades in out-of-distribution scenarios. We conclude that this generalization gap is a critical barrier for practitioners, as it severely limits the practical use of pre-trained approaches for real-world applications. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2509_19849 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Analyzing Generalization in Pre-Trained Symbolic Regression Voigt, Henrik Kahlmeyer, Paul Lawonn, Kai Habeck, Michael Giesen, Joachim Machine Learning Artificial Intelligence Symbolic regression algorithms search a space of mathematical expressions for formulas that explain given data. Transformer-based models have emerged as a promising, scalable approach shifting the expensive combinatorial search to a large-scale pre-training phase. However, the success of these models is critically dependent on their pre-training data. Their ability to generalize to problems outside of this pre-training distribution remains largely unexplored. In this work, we conduct a systematic empirical study to evaluate the generalization capabilities of pre-trained, transformer-based symbolic regression. We rigorously test performance both within the pre-training distribution and on a series of out-of-distribution challenges for several state of the art approaches. Our findings reveal a significant dichotomy: while pre-trained models perform well in-distribution, the performance consistently degrades in out-of-distribution scenarios. We conclude that this generalization gap is a critical barrier for practitioners, as it severely limits the practical use of pre-trained approaches for real-world applications. |
| title | Analyzing Generalization in Pre-Trained Symbolic Regression |
| topic | Machine Learning Artificial Intelligence |
| url | https://arxiv.org/abs/2509.19849 |