Masked Particle Modeling on Sets: Towards Self-Supervised High Energy Physics Foundation Models
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2024
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866913426089967616 |
|---|---|
| author | Golling, Tobias Heinrich, Lukas Kagan, Michael Klein, Samuel Leigh, Matthew Osadchy, Margarita Raine, John Andrew |
| author_facet | Golling, Tobias Heinrich, Lukas Kagan, Michael Klein, Samuel Leigh, Matthew Osadchy, Margarita Raine, John Andrew |
| contents | We propose masked particle modeling (MPM) as a self-supervised method for learning generic, transferable, and reusable representations on unordered sets of inputs for use in high energy physics (HEP) scientific data. This work provides a novel scheme to perform masked modeling based pre-training to learn permutation invariant functions on sets. More generally, this work provides a step towards building large foundation models for HEP that can be generically pre-trained with self-supervised learning and later fine-tuned for a variety of down-stream tasks. In MPM, particles in a set are masked and the training objective is to recover their identity, as defined by a discretized token representation of a pre-trained vector quantized variational autoencoder. We study the efficacy of the method in samples of high energy jets at collider physics experiments, including studies on the impact of discretization, permutation invariance, and ordering. We also study the fine-tuning capability of the model, showing that it can be adapted to tasks such as supervised and weakly supervised jet classification, and that the model can transfer efficiently with small fine-tuning data sets to new classes and new data domains. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2401_13537 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Masked Particle Modeling on Sets: Towards Self-Supervised High Energy Physics Foundation Models Golling, Tobias Heinrich, Lukas Kagan, Michael Klein, Samuel Leigh, Matthew Osadchy, Margarita Raine, John Andrew High Energy Physics - Phenomenology Machine Learning High Energy Physics - Experiment Data Analysis, Statistics and Probability We propose masked particle modeling (MPM) as a self-supervised method for learning generic, transferable, and reusable representations on unordered sets of inputs for use in high energy physics (HEP) scientific data. This work provides a novel scheme to perform masked modeling based pre-training to learn permutation invariant functions on sets. More generally, this work provides a step towards building large foundation models for HEP that can be generically pre-trained with self-supervised learning and later fine-tuned for a variety of down-stream tasks. In MPM, particles in a set are masked and the training objective is to recover their identity, as defined by a discretized token representation of a pre-trained vector quantized variational autoencoder. We study the efficacy of the method in samples of high energy jets at collider physics experiments, including studies on the impact of discretization, permutation invariance, and ordering. We also study the fine-tuning capability of the model, showing that it can be adapted to tasks such as supervised and weakly supervised jet classification, and that the model can transfer efficiently with small fine-tuning data sets to new classes and new data domains. |
| title | Masked Particle Modeling on Sets: Towards Self-Supervised High Energy Physics Foundation Models |
| topic | High Energy Physics - Phenomenology Machine Learning High Energy Physics - Experiment Data Analysis, Statistics and Probability |
| url | https://arxiv.org/abs/2401.13537 |