Composer: A Search Framework for Hybrid Neural Architecture Design
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2025
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866915851486101504 |
|---|---|
| author | Acun, Bilge Sinha, Prasoon Ardalani, Newsha Bae, Sangmin Golden, Alicia Lin, Chien-Yu Madhyastha, Meghana Sun, Fei Yadwadkar, Neeraja J. Wu, Carole-Jean |
| author_facet | Acun, Bilge Sinha, Prasoon Ardalani, Newsha Bae, Sangmin Golden, Alicia Lin, Chien-Yu Madhyastha, Meghana Sun, Fei Yadwadkar, Neeraja J. Wu, Carole-Jean |
| contents | Hybrid model architectures that combine computational primitives (e.g., Attention, MLP) in different ratios have shown promising performance beyond Transformers. Some studies have shown that different interleavings of primitives can affect model quality as well. However, prior works explore the hybrid model architecture design space manually. Due to the large design space and training costs, discovering hybrid models that combine key computational primitives for pre-training is challenging. In this work, we take a principled approach in designing a modular hybrid model architecture search framework -- Composer. Composer explores model architectures at a small scale and extrapolates the top-performing model architectures to a larger scale using our proposed scaling strategies. Using Composer, we discover new hybrid LLM architectures that outperform Llama 3.2. Compared to Llama 3.2 and previous state-of-the-art baselines, the new model architectures consistently reduce validation loss at parameter scales of 350M-3B and improve evaluation accuracy on the downstream tasks by up to 2.8-8.3% (1.1-3.1% on average) while improving both training and inference efficiency. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2510_00379 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Composer: A Search Framework for Hybrid Neural Architecture Design Acun, Bilge Sinha, Prasoon Ardalani, Newsha Bae, Sangmin Golden, Alicia Lin, Chien-Yu Madhyastha, Meghana Sun, Fei Yadwadkar, Neeraja J. Wu, Carole-Jean Machine Learning Hybrid model architectures that combine computational primitives (e.g., Attention, MLP) in different ratios have shown promising performance beyond Transformers. Some studies have shown that different interleavings of primitives can affect model quality as well. However, prior works explore the hybrid model architecture design space manually. Due to the large design space and training costs, discovering hybrid models that combine key computational primitives for pre-training is challenging. In this work, we take a principled approach in designing a modular hybrid model architecture search framework -- Composer. Composer explores model architectures at a small scale and extrapolates the top-performing model architectures to a larger scale using our proposed scaling strategies. Using Composer, we discover new hybrid LLM architectures that outperform Llama 3.2. Compared to Llama 3.2 and previous state-of-the-art baselines, the new model architectures consistently reduce validation loss at parameter scales of 350M-3B and improve evaluation accuracy on the downstream tasks by up to 2.8-8.3% (1.1-3.1% on average) while improving both training and inference efficiency. |
| title | Composer: A Search Framework for Hybrid Neural Architecture Design |
| topic | Machine Learning |
| url | https://arxiv.org/abs/2510.00379 |