Transferrable Surrogates in Expressive Neural Architecture Search Spaces
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2025
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866913924676321280 |
|---|---|
| author | Qin, Shiwen Kadlecová, Gabriela Pilát, Martin Cohen, Shay B. Neruda, Roman Crowley, Elliot J. Lukasik, Jovita Ericsson, Linus |
| author_facet | Qin, Shiwen Kadlecová, Gabriela Pilát, Martin Cohen, Shay B. Neruda, Roman Crowley, Elliot J. Lukasik, Jovita Ericsson, Linus |
| contents | Neural architecture search (NAS) faces a challenge in balancing the exploration of expressive, broad search spaces that enable architectural innovation with the need for efficient evaluation of architectures to effectively search such spaces. We investigate surrogate model training for improving search in highly expressive NAS search spaces based on context-free grammars. We show that i) surrogate models trained either using zero-cost-proxy metrics and neural graph features (GRAF) or by fine-tuning an off-the-shelf LM have high predictive power for the performance of architectures both within and across datasets, ii) these surrogates can be used to filter out bad architectures when searching on novel datasets, thereby significantly speeding up search and achieving better final performances, and iii) the surrogates can be further used directly as the search objective for huge speed-ups. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2504_12971 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Transferrable Surrogates in Expressive Neural Architecture Search Spaces Qin, Shiwen Kadlecová, Gabriela Pilát, Martin Cohen, Shay B. Neruda, Roman Crowley, Elliot J. Lukasik, Jovita Ericsson, Linus Machine Learning Artificial Intelligence Neural architecture search (NAS) faces a challenge in balancing the exploration of expressive, broad search spaces that enable architectural innovation with the need for efficient evaluation of architectures to effectively search such spaces. We investigate surrogate model training for improving search in highly expressive NAS search spaces based on context-free grammars. We show that i) surrogate models trained either using zero-cost-proxy metrics and neural graph features (GRAF) or by fine-tuning an off-the-shelf LM have high predictive power for the performance of architectures both within and across datasets, ii) these surrogates can be used to filter out bad architectures when searching on novel datasets, thereby significantly speeding up search and achieving better final performances, and iii) the surrogates can be further used directly as the search objective for huge speed-ups. |
| title | Transferrable Surrogates in Expressive Neural Architecture Search Spaces |
| topic | Machine Learning Artificial Intelligence |
| url | https://arxiv.org/abs/2504.12971 |