Falcon2-11B Technical Report
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , , , , , , , , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2024
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866914879924862976 |
|---|---|
| author | Malartic, Quentin Chowdhury, Nilabhra Roy Cojocaru, Ruxandra Farooq, Mugariya Campesan, Giulia Djilali, Yasser Abdelaziz Dahou Narayan, Sanath Singh, Ankit Velikanov, Maksim Boussaha, Basma El Amel Al-Yafeai, Mohammed Alobeidli, Hamza Qadi, Leen Al Seddik, Mohamed El Amine Fedyanin, Kirill Alami, Reda Hacid, Hakim |
| author_facet | Malartic, Quentin Chowdhury, Nilabhra Roy Cojocaru, Ruxandra Farooq, Mugariya Campesan, Giulia Djilali, Yasser Abdelaziz Dahou Narayan, Sanath Singh, Ankit Velikanov, Maksim Boussaha, Basma El Amel Al-Yafeai, Mohammed Alobeidli, Hamza Qadi, Leen Al Seddik, Mohamed El Amine Fedyanin, Kirill Alami, Reda Hacid, Hakim |
| contents | We introduce Falcon2-11B, a foundation model trained on over five trillion tokens, and its multimodal counterpart, Falcon2-11B-vlm, which is a vision-to-text model. We report our findings during the training of the Falcon2-11B which follows a multi-stage approach where the early stages are distinguished by their context length and a final stage where we use a curated, high-quality dataset. Additionally, we report the effect of doubling the batch size mid-training and how training loss spikes are affected by the learning rate. The downstream performance of the foundation model is evaluated on established benchmarks, including multilingual and code datasets. The foundation model shows strong generalization across all the tasks which makes it suitable for downstream finetuning use cases. For the vision language model, we report the performance on several benchmarks and show that our model achieves a higher average score compared to open-source models of similar size. The model weights and code of both Falcon2-11B and Falcon2-11B-vlm are made available under a permissive license. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2407_14885 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Falcon2-11B Technical Report Malartic, Quentin Chowdhury, Nilabhra Roy Cojocaru, Ruxandra Farooq, Mugariya Campesan, Giulia Djilali, Yasser Abdelaziz Dahou Narayan, Sanath Singh, Ankit Velikanov, Maksim Boussaha, Basma El Amel Al-Yafeai, Mohammed Alobeidli, Hamza Qadi, Leen Al Seddik, Mohamed El Amine Fedyanin, Kirill Alami, Reda Hacid, Hakim Computation and Language Computer Vision and Pattern Recognition We introduce Falcon2-11B, a foundation model trained on over five trillion tokens, and its multimodal counterpart, Falcon2-11B-vlm, which is a vision-to-text model. We report our findings during the training of the Falcon2-11B which follows a multi-stage approach where the early stages are distinguished by their context length and a final stage where we use a curated, high-quality dataset. Additionally, we report the effect of doubling the batch size mid-training and how training loss spikes are affected by the learning rate. The downstream performance of the foundation model is evaluated on established benchmarks, including multilingual and code datasets. The foundation model shows strong generalization across all the tasks which makes it suitable for downstream finetuning use cases. For the vision language model, we report the performance on several benchmarks and show that our model achieves a higher average score compared to open-source models of similar size. The model weights and code of both Falcon2-11B and Falcon2-11B-vlm are made available under a permissive license. |
| title | Falcon2-11B Technical Report |
| topic | Computation and Language Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2407.14885 |