FFN Fusion: Rethinking Sequential Computation in Large Language Models
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866915212049776640 |
|---|---|
| author | Bercovich, Akhiad Dabbah, Mohammad Puny, Omri Galil, Ido Geifman, Amnon Geifman, Yonatan Golan, Izhak Karpas, Ehud Levy, Itay Moshe, Zach Nabwani, Najeeb Ronen, Tomer Schen, Itamar Segal, Elad Shahaf, Ido Tropp, Oren Zilberstein, Ran El-Yaniv, Ran |
| author_facet | Bercovich, Akhiad Dabbah, Mohammad Puny, Omri Galil, Ido Geifman, Amnon Geifman, Yonatan Golan, Izhak Karpas, Ehud Levy, Itay Moshe, Zach Nabwani, Najeeb Ronen, Tomer Schen, Itamar Segal, Elad Shahaf, Ido Tropp, Oren Zilberstein, Ran El-Yaniv, Ran |
| contents | We introduce FFN Fusion, an architectural optimization technique that reduces sequential computation in large language models by identifying and exploiting natural opportunities for parallelization. Our key insight is that sequences of Feed-Forward Network (FFN) layers, particularly those remaining after the removal of specific attention layers, can often be parallelized with minimal accuracy impact. We develop a principled methodology for identifying and fusing such sequences, transforming them into parallel operations that significantly reduce inference latency while preserving model behavior. Applying these techniques to Llama-3.1-405B-Instruct, we create Llama-Nemotron-Ultra-253B-Base (Ultra-253B-Base), an efficient and soon-to-be publicly available model that achieves a 1.71X speedup in inference latency and 35X lower per-token cost while maintaining strong performance across benchmarks. Through extensive experiments on models from 49B to 253B parameters, we demonstrate that FFN Fusion becomes increasingly effective at larger scales and can complement existing optimization techniques like quantization and pruning. Most intriguingly, we find that even full transformer blocks containing both attention and FFN layers can sometimes be parallelized, suggesting new directions for neural architecture design. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2503_18908 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | FFN Fusion: Rethinking Sequential Computation in Large Language Models Bercovich, Akhiad Dabbah, Mohammad Puny, Omri Galil, Ido Geifman, Amnon Geifman, Yonatan Golan, Izhak Karpas, Ehud Levy, Itay Moshe, Zach Nabwani, Najeeb Ronen, Tomer Schen, Itamar Segal, Elad Shahaf, Ido Tropp, Oren Zilberstein, Ran El-Yaniv, Ran Machine Learning We introduce FFN Fusion, an architectural optimization technique that reduces sequential computation in large language models by identifying and exploiting natural opportunities for parallelization. Our key insight is that sequences of Feed-Forward Network (FFN) layers, particularly those remaining after the removal of specific attention layers, can often be parallelized with minimal accuracy impact. We develop a principled methodology for identifying and fusing such sequences, transforming them into parallel operations that significantly reduce inference latency while preserving model behavior. Applying these techniques to Llama-3.1-405B-Instruct, we create Llama-Nemotron-Ultra-253B-Base (Ultra-253B-Base), an efficient and soon-to-be publicly available model that achieves a 1.71X speedup in inference latency and 35X lower per-token cost while maintaining strong performance across benchmarks. Through extensive experiments on models from 49B to 253B parameters, we demonstrate that FFN Fusion becomes increasingly effective at larger scales and can complement existing optimization techniques like quantization and pruning. Most intriguingly, we find that even full transformer blocks containing both attention and FFN layers can sometimes be parallelized, suggesting new directions for neural architecture design. |
| title | FFN Fusion: Rethinking Sequential Computation in Large Language Models |
| topic | Machine Learning |
| url | https://arxiv.org/abs/2503.18908 |