multivariateGPT: a decoder-only transformer for multivariate categorical and numeric data
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2025
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866913866459381760 |
|---|---|
| author | Loza, Andrew J. Kim, Jun Yup Song, Shangzheng Liu, Yihang Sung, Joseph J. Y. Taylor, R Andrew Shung, Dennis L. |
| author_facet | Loza, Andrew J. Kim, Jun Yup Song, Shangzheng Liu, Yihang Sung, Joseph J. Y. Taylor, R Andrew Shung, Dennis L. |
| contents | Real-world processes often generate data that are a mix of categorical and numeric values that are recorded at irregular and informative intervals. Discrete token-based approaches are limited in numeric representation capacity while methods like neural ordinary differential equations are not well suited for categorical data or informative sampling and require augmentation to handle certain classes of trajectories. Here, we present multivariateGPT, a single architecture for modeling sequences of mixed categorical (including tokenized text) and numeric data. This is accomplished with an autoregressive sequence decomposition, embedding scheme, and loss function that extend the next token prediction task to likelihood estimation of the joint distribution of next token class and value. We demonstrate how this approach can efficiently learn to generalize patterns in simple physical systems and model complex time series including electrocardiograms and multivariate electronic health record data. This work extends the utility of transformer based models to additional classes of data. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2505_21680 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | multivariateGPT: a decoder-only transformer for multivariate categorical and numeric data Loza, Andrew J. Kim, Jun Yup Song, Shangzheng Liu, Yihang Sung, Joseph J. Y. Taylor, R Andrew Shung, Dennis L. Machine Learning Artificial Intelligence 68T07 I.2.6; I.5.1 Real-world processes often generate data that are a mix of categorical and numeric values that are recorded at irregular and informative intervals. Discrete token-based approaches are limited in numeric representation capacity while methods like neural ordinary differential equations are not well suited for categorical data or informative sampling and require augmentation to handle certain classes of trajectories. Here, we present multivariateGPT, a single architecture for modeling sequences of mixed categorical (including tokenized text) and numeric data. This is accomplished with an autoregressive sequence decomposition, embedding scheme, and loss function that extend the next token prediction task to likelihood estimation of the joint distribution of next token class and value. We demonstrate how this approach can efficiently learn to generalize patterns in simple physical systems and model complex time series including electrocardiograms and multivariate electronic health record data. This work extends the utility of transformer based models to additional classes of data. |
| title | multivariateGPT: a decoder-only transformer for multivariate categorical and numeric data |
| topic | Machine Learning Artificial Intelligence 68T07 I.2.6; I.5.1 |
| url | https://arxiv.org/abs/2505.21680 |