Saved in:
| Main Authors: | , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2506.05641 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866909640916205568 |
|---|---|
| author | Zhmoginov, Andrey Lee, Jihwan Sandler, Mark |
| author_facet | Zhmoginov, Andrey Lee, Jihwan Sandler, Mark |
| contents | Modern Foundation Models (FMs) are typically trained on corpora spanning a wide range of different data modalities, topics and downstream tasks. Utilizing these models can be very computationally expensive and is out of reach for most consumer devices. Furthermore, most of the broad FM knowledge may actually be irrelevant for a specific task at hand. Here we explore a technique for mapping parameters of a large Transformer to parameters of a smaller specialized model. By making this transformation task-specific, we aim to capture a narrower scope of the knowledge needed for performing a specific task by a smaller model. We study our method on image modeling tasks, showing that performance of generated models exceeds that of universal conditional models. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2506_05641 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Projectable Models: One-Shot Generation of Small Specialized Transformers from Large Ones Zhmoginov, Andrey Lee, Jihwan Sandler, Mark Machine Learning Computation and Language Modern Foundation Models (FMs) are typically trained on corpora spanning a wide range of different data modalities, topics and downstream tasks. Utilizing these models can be very computationally expensive and is out of reach for most consumer devices. Furthermore, most of the broad FM knowledge may actually be irrelevant for a specific task at hand. Here we explore a technique for mapping parameters of a large Transformer to parameters of a smaller specialized model. By making this transformation task-specific, we aim to capture a narrower scope of the knowledge needed for performing a specific task by a smaller model. We study our method on image modeling tasks, showing that performance of generated models exceeds that of universal conditional models. |
| title | Projectable Models: One-Shot Generation of Small Specialized Transformers from Large Ones |
| topic | Machine Learning Computation and Language |
| url | https://arxiv.org/abs/2506.05641 |