Is Tokenization Needed for Masked Particle Modelling?
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866912052529856512 |
|---|---|
| author | Leigh, Matthew Klein, Samuel Charton, François Golling, Tobias Heinrich, Lukas Kagan, Michael Ochoa, Inês Osadchy, Margarita |
| author_facet | Leigh, Matthew Klein, Samuel Charton, François Golling, Tobias Heinrich, Lukas Kagan, Michael Ochoa, Inês Osadchy, Margarita |
| contents | In this work, we significantly enhance masked particle modeling (MPM), a self-supervised learning scheme for constructing highly expressive representations of unordered sets relevant to developing foundation models for high-energy physics. In MPM, a model is trained to recover the missing elements of a set, a learning objective that requires no labels and can be applied directly to experimental data. We achieve significant performance improvements over previous work on MPM by addressing inefficiencies in the implementation and incorporating a more powerful decoder. We compare several pre-training tasks and introduce new reconstruction methods that utilize conditional generative models without data tokenization or discretization. We show that these new methods outperform the tokenized learning objective from the original MPM on a new test bed for foundation models for jets, which includes using a wide variety of downstream tasks relevant to jet physics, such as classification, secondary vertex finding, and track identification. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2409_12589 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Is Tokenization Needed for Masked Particle Modelling? Leigh, Matthew Klein, Samuel Charton, François Golling, Tobias Heinrich, Lukas Kagan, Michael Ochoa, Inês Osadchy, Margarita High Energy Physics - Phenomenology Machine Learning In this work, we significantly enhance masked particle modeling (MPM), a self-supervised learning scheme for constructing highly expressive representations of unordered sets relevant to developing foundation models for high-energy physics. In MPM, a model is trained to recover the missing elements of a set, a learning objective that requires no labels and can be applied directly to experimental data. We achieve significant performance improvements over previous work on MPM by addressing inefficiencies in the implementation and incorporating a more powerful decoder. We compare several pre-training tasks and introduce new reconstruction methods that utilize conditional generative models without data tokenization or discretization. We show that these new methods outperform the tokenized learning objective from the original MPM on a new test bed for foundation models for jets, which includes using a wide variety of downstream tasks relevant to jet physics, such as classification, secondary vertex finding, and track identification. |
| title | Is Tokenization Needed for Masked Particle Modelling? |
| topic | High Energy Physics - Phenomenology Machine Learning |
| url | https://arxiv.org/abs/2409.12589 |