Distilling Tabular Foundation Models for Structured Health Data
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2026
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866910233041829888 |
|---|---|
| author | Tanna, Aditya Bouarour, Nassim Bouadi, Mohamed Sankarapu, Vinay Kumar Seth, Pratinav |
| author_facet | Tanna, Aditya Bouarour, Nassim Bouadi, Mohamed Sankarapu, Vinay Kumar Seth, Pratinav |
| contents | Tabular foundation models (TFMs) achieve strong performance on health datasets, but their inference cost and infrastructure requirements limit practical use. We study whether their predictive behavior can be transferred to lightweight tabular models through knowledge distillation. Since in-context TFMs condition on the training set at inference time, naive distillation can introduce context leakage; we address this with stratified out-of-fold teacher labeling. Across $19$ healthcare datasets, $6$ TFM teachers, $4$ student families, and several multi-teacher ensembles, we find that distilled students retain at least $90\%$ of teacher AUC, outperforming teachers in some cases, while running at least $26\times$ faster on CPU and preserving calibration and fairness critical for health applications. Moreover, multi-teacher averaging does not consistently improve over the best single teacher. Leakage-aware distillation is thus a viable route for bringing TFM-quality predictions into inference-constrained health settings. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2605_18702 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | Distilling Tabular Foundation Models for Structured Health Data Tanna, Aditya Bouarour, Nassim Bouadi, Mohamed Sankarapu, Vinay Kumar Seth, Pratinav Machine Learning Artificial Intelligence Tabular foundation models (TFMs) achieve strong performance on health datasets, but their inference cost and infrastructure requirements limit practical use. We study whether their predictive behavior can be transferred to lightweight tabular models through knowledge distillation. Since in-context TFMs condition on the training set at inference time, naive distillation can introduce context leakage; we address this with stratified out-of-fold teacher labeling. Across $19$ healthcare datasets, $6$ TFM teachers, $4$ student families, and several multi-teacher ensembles, we find that distilled students retain at least $90\%$ of teacher AUC, outperforming teachers in some cases, while running at least $26\times$ faster on CPU and preserving calibration and fairness critical for health applications. Moreover, multi-teacher averaging does not consistently improve over the best single teacher. Leakage-aware distillation is thus a viable route for bringing TFM-quality predictions into inference-constrained health settings. |
| title | Distilling Tabular Foundation Models for Structured Health Data |
| topic | Machine Learning Artificial Intelligence |
| url | https://arxiv.org/abs/2605.18702 |