Saved in:
| Main Authors: | , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2512.04149 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866909943068622848 |
|---|---|
| author | Birk, Joschka Hallin, Anna Kasieczka, Gregor Madzharova, Nikol Pang, Ian Shih, David |
| author_facet | Birk, Joschka Hallin, Anna Kasieczka, Gregor Madzharova, Nikol Pang, Ian Shih, David |
| contents | Next token prediction is an attractive pre-training task for jet foundation models, in that it is simulation free and enables excellent generative capabilities that can transfer across datasets. Here we study multiple improvements to next token prediction, building on the initial work of OmniJet-$α$. Instead of tokenizing particles and subsequently only using the token-ID as the model input for both the generative and the classification task, we adopt a hybrid setup, which allows us to use continuous feature vectors as model input while only using token-IDs in the next token prediction target. Secondly, we explore a combined pre-training strategy that combines masked particle modeling and generative learning objectives. Taken together, these changes greatly improve the performance in downstream classification tasks without any loss in generative performance. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2512_04149 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Enhancing next token prediction based pre-training for jet foundation models Birk, Joschka Hallin, Anna Kasieczka, Gregor Madzharova, Nikol Pang, Ian Shih, David High Energy Physics - Phenomenology Machine Learning High Energy Physics - Experiment Data Analysis, Statistics and Probability Next token prediction is an attractive pre-training task for jet foundation models, in that it is simulation free and enables excellent generative capabilities that can transfer across datasets. Here we study multiple improvements to next token prediction, building on the initial work of OmniJet-$α$. Instead of tokenizing particles and subsequently only using the token-ID as the model input for both the generative and the classification task, we adopt a hybrid setup, which allows us to use continuous feature vectors as model input while only using token-IDs in the next token prediction target. Secondly, we explore a combined pre-training strategy that combines masked particle modeling and generative learning objectives. Taken together, these changes greatly improve the performance in downstream classification tasks without any loss in generative performance. |
| title | Enhancing next token prediction based pre-training for jet foundation models |
| topic | High Energy Physics - Phenomenology Machine Learning High Energy Physics - Experiment Data Analysis, Statistics and Probability |
| url | https://arxiv.org/abs/2512.04149 |