A Simple Baseline for Predicting Events with Auto-Regressive Tabular Transformers
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866915001422315520 |
|---|---|
| author | Stein, Alex Sharpe, Samuel Bergman, Doron Kumar, Senthil Bruss, C. Bayan Dickerson, John Goldstein, Tom Goldblum, Micah |
| author_facet | Stein, Alex Sharpe, Samuel Bergman, Doron Kumar, Senthil Bruss, C. Bayan Dickerson, John Goldstein, Tom Goldblum, Micah |
| contents | Many real-world applications of tabular data involve using historic events to predict properties of new ones, for example whether a credit card transaction is fraudulent or what rating a customer will assign a product on a retail platform. Existing approaches to event prediction include costly, brittle, and application-dependent techniques such as time-aware positional embeddings, learned row and field encodings, and oversampling methods for addressing class imbalance. Moreover, these approaches often assume specific use-cases, for example that we know the labels of all historic events or that we only predict a pre-specified label and not the data's features themselves. In this work, we propose a simple but flexible baseline using standard autoregressive LLM-style transformers with elementary positional embeddings and a causal language modeling objective. Our baseline outperforms existing approaches across popular datasets and can be employed for various use-cases. We demonstrate that the same model can predict labels, impute missing values, or model event sequences. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2410_10648 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | A Simple Baseline for Predicting Events with Auto-Regressive Tabular Transformers Stein, Alex Sharpe, Samuel Bergman, Doron Kumar, Senthil Bruss, C. Bayan Dickerson, John Goldstein, Tom Goldblum, Micah Machine Learning Computational Engineering, Finance, and Science Many real-world applications of tabular data involve using historic events to predict properties of new ones, for example whether a credit card transaction is fraudulent or what rating a customer will assign a product on a retail platform. Existing approaches to event prediction include costly, brittle, and application-dependent techniques such as time-aware positional embeddings, learned row and field encodings, and oversampling methods for addressing class imbalance. Moreover, these approaches often assume specific use-cases, for example that we know the labels of all historic events or that we only predict a pre-specified label and not the data's features themselves. In this work, we propose a simple but flexible baseline using standard autoregressive LLM-style transformers with elementary positional embeddings and a causal language modeling objective. Our baseline outperforms existing approaches across popular datasets and can be employed for various use-cases. We demonstrate that the same model can predict labels, impute missing values, or model event sequences. |
| title | A Simple Baseline for Predicting Events with Auto-Regressive Tabular Transformers |
| topic | Machine Learning Computational Engineering, Finance, and Science |
| url | https://arxiv.org/abs/2410.10648 |