In-Context Learning of Physical Properties: Few-Shot Adaptation to Out-of-Distribution Molecular Graphs
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2024
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866910470391201792 |
|---|---|
| author | Kaszuba, Grzegorz Naghdi, Amirhossein D. Massa, Dario Papanikolaou, Stefanos Jaszkiewicz, Andrzej Sankowski, Piotr |
| author_facet | Kaszuba, Grzegorz Naghdi, Amirhossein D. Massa, Dario Papanikolaou, Stefanos Jaszkiewicz, Andrzej Sankowski, Piotr |
| contents | Large language models manifest the ability of few-shot adaptation to a sequence of provided examples. This behavior, known as in-context learning, allows for performing nontrivial machine learning tasks during inference only. In this work, we address the question: can we leverage in-context learning to predict out-of-distribution materials properties? However, this would not be possible for structure property prediction tasks unless an effective method is found to pass atomic-level geometric features to the transformer model. To address this problem, we employ a compound model in which GPT-2 acts on the output of geometry-aware graph neural networks to adapt in-context information. To demonstrate our model's capabilities, we partition the QM9 dataset into sequences of molecules that share a common substructure and use them for in-context learning. This approach significantly improves the performance of the model on out-of-distribution examples, surpassing the one of general graph neural network models. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2406_01808 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | In-Context Learning of Physical Properties: Few-Shot Adaptation to Out-of-Distribution Molecular Graphs Kaszuba, Grzegorz Naghdi, Amirhossein D. Massa, Dario Papanikolaou, Stefanos Jaszkiewicz, Andrzej Sankowski, Piotr Machine Learning Materials Science Large language models manifest the ability of few-shot adaptation to a sequence of provided examples. This behavior, known as in-context learning, allows for performing nontrivial machine learning tasks during inference only. In this work, we address the question: can we leverage in-context learning to predict out-of-distribution materials properties? However, this would not be possible for structure property prediction tasks unless an effective method is found to pass atomic-level geometric features to the transformer model. To address this problem, we employ a compound model in which GPT-2 acts on the output of geometry-aware graph neural networks to adapt in-context information. To demonstrate our model's capabilities, we partition the QM9 dataset into sequences of molecules that share a common substructure and use them for in-context learning. This approach significantly improves the performance of the model on out-of-distribution examples, surpassing the one of general graph neural network models. |
| title | In-Context Learning of Physical Properties: Few-Shot Adaptation to Out-of-Distribution Molecular Graphs |
| topic | Machine Learning Materials Science |
| url | https://arxiv.org/abs/2406.01808 |