DataRater: Meta-Learned Dataset Curation
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2025
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866908613593792512 |
|---|---|
| author | Calian, Dan A. Farquhar, Gregory Kemaev, Iurii Zintgraf, Luisa M. Hessel, Matteo Shar, Jeremy Oh, Junhyuk György, András Schaul, Tom Dean, Jeffrey van Hasselt, Hado Silver, David |
| author_facet | Calian, Dan A. Farquhar, Gregory Kemaev, Iurii Zintgraf, Luisa M. Hessel, Matteo Shar, Jeremy Oh, Junhyuk György, András Schaul, Tom Dean, Jeffrey van Hasselt, Hado Silver, David |
| contents | The quality of foundation models depends heavily on their training data. Consequently, great efforts have been put into dataset curation. Yet most approaches rely on manual tuning of coarse-grained mixtures of large buckets of data, or filtering by hand-crafted heuristics. An approach that is ultimately more scalable (let alone more satisfying) is to \emph{learn} which data is actually valuable for training. This type of meta-learning could allow more sophisticated, fine-grained, and effective curation. Our proposed \emph{DataRater} is an instance of this idea. It estimates the value of training on any particular data point. This is done by meta-learning using `meta-gradients', with the objective of improving training efficiency on held out data. In extensive experiments across a range of model scales and datasets, we find that using our DataRater to filter data is highly effective, resulting in significantly improved compute efficiency. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2505_17895 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | DataRater: Meta-Learned Dataset Curation Calian, Dan A. Farquhar, Gregory Kemaev, Iurii Zintgraf, Luisa M. Hessel, Matteo Shar, Jeremy Oh, Junhyuk György, András Schaul, Tom Dean, Jeffrey van Hasselt, Hado Silver, David Machine Learning Artificial Intelligence I.2.6 The quality of foundation models depends heavily on their training data. Consequently, great efforts have been put into dataset curation. Yet most approaches rely on manual tuning of coarse-grained mixtures of large buckets of data, or filtering by hand-crafted heuristics. An approach that is ultimately more scalable (let alone more satisfying) is to \emph{learn} which data is actually valuable for training. This type of meta-learning could allow more sophisticated, fine-grained, and effective curation. Our proposed \emph{DataRater} is an instance of this idea. It estimates the value of training on any particular data point. This is done by meta-learning using `meta-gradients', with the objective of improving training efficiency on held out data. In extensive experiments across a range of model scales and datasets, we find that using our DataRater to filter data is highly effective, resulting in significantly improved compute efficiency. |
| title | DataRater: Meta-Learned Dataset Curation |
| topic | Machine Learning Artificial Intelligence I.2.6 |
| url | https://arxiv.org/abs/2505.17895 |