Falcon-H1: A Family of Hybrid-Head Language Models Redefining Efficiency and Performance
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , , , , , , , , , , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2025
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866915417605275648 |
|---|---|
| author | Zuo, Jingwei Velikanov, Maksim Chahed, Ilyas Belkada, Younes Rhayem, Dhia Eddine Kunsch, Guillaume Hacid, Hakim Yous, Hamza Farhat, Brahim Khadraoui, Ibrahim Farooq, Mugariya Campesan, Giulia Cojocaru, Ruxandra Djilali, Yasser Hu, Shi Chaabane, Iheb Khanna, Puneesh Seddik, Mohamed El Amine Huynh, Ngoc Dung Khac, Phuc Le AlQadi, Leen Mokeddem, Billel Chami, Mohamed Abubaker, Abdalgader Lubinets, Mikhail Piskorski, Kacper Frikha, Slim |
| author_facet | Zuo, Jingwei Velikanov, Maksim Chahed, Ilyas Belkada, Younes Rhayem, Dhia Eddine Kunsch, Guillaume Hacid, Hakim Yous, Hamza Farhat, Brahim Khadraoui, Ibrahim Farooq, Mugariya Campesan, Giulia Cojocaru, Ruxandra Djilali, Yasser Hu, Shi Chaabane, Iheb Khanna, Puneesh Seddik, Mohamed El Amine Huynh, Ngoc Dung Khac, Phuc Le AlQadi, Leen Mokeddem, Billel Chami, Mohamed Abubaker, Abdalgader Lubinets, Mikhail Piskorski, Kacper Frikha, Slim |
| contents | In this report, we introduce Falcon-H1, a new series of large language models (LLMs) featuring hybrid architecture designs optimized for both high performance and efficiency across diverse use cases. Unlike earlier Falcon models built solely on Transformer or Mamba architectures, Falcon-H1 adopts a parallel hybrid approach that combines Transformer-based attention with State Space Models (SSMs), known for superior long-context memory and computational efficiency. We systematically revisited model design, data strategy, and training dynamics, challenging conventional practices in the field. Falcon-H1 is released in multiple configurations, including base and instruction-tuned variants at 0.5B, 1.5B, 1.5B-deep, 3B, 7B, and 34B parameters. Quantized instruction-tuned models are also available, totaling over 30 checkpoints on Hugging Face Hub. Falcon-H1 models demonstrate state-of-the-art performance and exceptional parameter and training efficiency. The flagship Falcon-H1-34B matches or outperforms models up to 70B scale, such as Qwen3-32B, Qwen2.5-72B, and Llama3.3-70B, while using fewer parameters and less data. Smaller models show similar trends: the Falcon-H1-1.5B-Deep rivals current leading 7B-10B models, and Falcon-H1-0.5B performs comparably to typical 7B models from 2024. These models excel across reasoning, mathematics, multilingual tasks, instruction following, and scientific knowledge. With support for up to 256K context tokens and 18 languages, Falcon-H1 is suitable for a wide range of applications. All models are released under a permissive open-source license, underscoring our commitment to accessible and impactful AI research. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2507_22448 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Falcon-H1: A Family of Hybrid-Head Language Models Redefining Efficiency and Performance Zuo, Jingwei Velikanov, Maksim Chahed, Ilyas Belkada, Younes Rhayem, Dhia Eddine Kunsch, Guillaume Hacid, Hakim Yous, Hamza Farhat, Brahim Khadraoui, Ibrahim Farooq, Mugariya Campesan, Giulia Cojocaru, Ruxandra Djilali, Yasser Hu, Shi Chaabane, Iheb Khanna, Puneesh Seddik, Mohamed El Amine Huynh, Ngoc Dung Khac, Phuc Le AlQadi, Leen Mokeddem, Billel Chami, Mohamed Abubaker, Abdalgader Lubinets, Mikhail Piskorski, Kacper Frikha, Slim Computation and Language In this report, we introduce Falcon-H1, a new series of large language models (LLMs) featuring hybrid architecture designs optimized for both high performance and efficiency across diverse use cases. Unlike earlier Falcon models built solely on Transformer or Mamba architectures, Falcon-H1 adopts a parallel hybrid approach that combines Transformer-based attention with State Space Models (SSMs), known for superior long-context memory and computational efficiency. We systematically revisited model design, data strategy, and training dynamics, challenging conventional practices in the field. Falcon-H1 is released in multiple configurations, including base and instruction-tuned variants at 0.5B, 1.5B, 1.5B-deep, 3B, 7B, and 34B parameters. Quantized instruction-tuned models are also available, totaling over 30 checkpoints on Hugging Face Hub. Falcon-H1 models demonstrate state-of-the-art performance and exceptional parameter and training efficiency. The flagship Falcon-H1-34B matches or outperforms models up to 70B scale, such as Qwen3-32B, Qwen2.5-72B, and Llama3.3-70B, while using fewer parameters and less data. Smaller models show similar trends: the Falcon-H1-1.5B-Deep rivals current leading 7B-10B models, and Falcon-H1-0.5B performs comparably to typical 7B models from 2024. These models excel across reasoning, mathematics, multilingual tasks, instruction following, and scientific knowledge. With support for up to 256K context tokens and 18 languages, Falcon-H1 is suitable for a wide range of applications. All models are released under a permissive open-source license, underscoring our commitment to accessible and impactful AI research. |
| title | Falcon-H1: A Family of Hybrid-Head Language Models Redefining Efficiency and Performance |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2507.22448 |