Making Pre-trained Language Models Great on Tabular Prediction
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866913261787545600 |
|---|---|
| author | Yan, Jiahuan Zheng, Bo Xu, Hongxia Zhu, Yiheng Chen, Danny Z. Sun, Jimeng Wu, Jian Chen, Jintai |
| author_facet | Yan, Jiahuan Zheng, Bo Xu, Hongxia Zhu, Yiheng Chen, Danny Z. Sun, Jimeng Wu, Jian Chen, Jintai |
| contents | The transferability of deep neural networks (DNNs) has made significant progress in image and language processing. However, due to the heterogeneity among tables, such DNN bonus is still far from being well exploited on tabular data prediction (e.g., regression or classification tasks). Condensing knowledge from diverse domains, language models (LMs) possess the capability to comprehend feature names from various tables, potentially serving as versatile learners in transferring knowledge across distinct tables and diverse prediction tasks, but their discrete text representation space is inherently incompatible with numerical feature values in tables. In this paper, we present TP-BERTa, a specifically pre-trained LM for tabular data prediction. Concretely, a novel relative magnitude tokenization converts scalar numerical feature values to finely discrete, high-dimensional tokens, and an intra-feature attention approach integrates feature values with the corresponding feature names. Comprehensive experiments demonstrate that our pre-trained TP-BERTa leads the performance among tabular DNNs and is competitive with Gradient Boosted Decision Tree models in typical tabular data regime. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2403_01841 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Making Pre-trained Language Models Great on Tabular Prediction Yan, Jiahuan Zheng, Bo Xu, Hongxia Zhu, Yiheng Chen, Danny Z. Sun, Jimeng Wu, Jian Chen, Jintai Computation and Language Machine Learning The transferability of deep neural networks (DNNs) has made significant progress in image and language processing. However, due to the heterogeneity among tables, such DNN bonus is still far from being well exploited on tabular data prediction (e.g., regression or classification tasks). Condensing knowledge from diverse domains, language models (LMs) possess the capability to comprehend feature names from various tables, potentially serving as versatile learners in transferring knowledge across distinct tables and diverse prediction tasks, but their discrete text representation space is inherently incompatible with numerical feature values in tables. In this paper, we present TP-BERTa, a specifically pre-trained LM for tabular data prediction. Concretely, a novel relative magnitude tokenization converts scalar numerical feature values to finely discrete, high-dimensional tokens, and an intra-feature attention approach integrates feature values with the corresponding feature names. Comprehensive experiments demonstrate that our pre-trained TP-BERTa leads the performance among tabular DNNs and is competitive with Gradient Boosted Decision Tree models in typical tabular data regime. |
| title | Making Pre-trained Language Models Great on Tabular Prediction |
| topic | Computation and Language Machine Learning |
| url | https://arxiv.org/abs/2403.01841 |