From Embeddings to Equations: Genetic-Programming Surrogates for Interpretable Transformer Classification
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866916970645946368 |
|---|---|
| author | Khorshidi, Mohammad Sadegh Yazdanjue, Navid Gharoun, Hassan Nikoo, Mohammad Reza Chen, Fang Gandomi, Amir H. |
| author_facet | Khorshidi, Mohammad Sadegh Yazdanjue, Navid Gharoun, Hassan Nikoo, Mohammad Reza Chen, Fang Gandomi, Amir H. |
| contents | We study symbolic surrogate modeling of frozen Transformer embeddings to obtain compact, auditable classifiers with calibrated probabilities. For five benchmarks (SST2G, 20NG, MNIST, CIFAR10, MSC17), embeddings from ModernBERT, DINOv2, and SigLIP are partitioned on the training set into disjoint, information-preserving views via semantic-preserving feature partitioning (SPFP). A cooperative multi-population genetic program (MEGP) then learns additive, closed-form logit programs over these views. Across 30 runs per dataset we report F1, AUC, log-loss, Brier, expected calibration error (ECE), and symbolic complexity; a canonical model is chosen by a one-standard-error rule on validation F1 with a parsimony tie-break. Temperature scaling fitted on validation yields substantial ECE reductions on test. The resulting surrogates achieve strong discrimination (up to F1 around 0.99 on MNIST, CIFAR10, MSC17; around 0.95 on SST2G), while 20NG remains most challenging. We provide reliability diagrams, dimension usage and overlap statistics, contribution-based importances, and global effect profiles (PDP and ALE), demonstrating faithful, cross-modal explanations grounded in explicit programs. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2509_21341 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | From Embeddings to Equations: Genetic-Programming Surrogates for Interpretable Transformer Classification Khorshidi, Mohammad Sadegh Yazdanjue, Navid Gharoun, Hassan Nikoo, Mohammad Reza Chen, Fang Gandomi, Amir H. Neural and Evolutionary Computing Artificial Intelligence Machine Learning 68T07 We study symbolic surrogate modeling of frozen Transformer embeddings to obtain compact, auditable classifiers with calibrated probabilities. For five benchmarks (SST2G, 20NG, MNIST, CIFAR10, MSC17), embeddings from ModernBERT, DINOv2, and SigLIP are partitioned on the training set into disjoint, information-preserving views via semantic-preserving feature partitioning (SPFP). A cooperative multi-population genetic program (MEGP) then learns additive, closed-form logit programs over these views. Across 30 runs per dataset we report F1, AUC, log-loss, Brier, expected calibration error (ECE), and symbolic complexity; a canonical model is chosen by a one-standard-error rule on validation F1 with a parsimony tie-break. Temperature scaling fitted on validation yields substantial ECE reductions on test. The resulting surrogates achieve strong discrimination (up to F1 around 0.99 on MNIST, CIFAR10, MSC17; around 0.95 on SST2G), while 20NG remains most challenging. We provide reliability diagrams, dimension usage and overlap statistics, contribution-based importances, and global effect profiles (PDP and ALE), demonstrating faithful, cross-modal explanations grounded in explicit programs. |
| title | From Embeddings to Equations: Genetic-Programming Surrogates for Interpretable Transformer Classification |
| topic | Neural and Evolutionary Computing Artificial Intelligence Machine Learning 68T07 |
| url | https://arxiv.org/abs/2509.21341 |