Saved in:
| Main Authors: | , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2509.22463 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866917004267487232 |
|---|---|
| author | Liu, Xinyu Li, Bei Liu, Jiahao Ruan, Junhao Jiao, Kechen Tang, Hongyin Wang, Jingang Tong, Xiao Zhu, Jingbo |
| author_facet | Liu, Xinyu Li, Bei Liu, Jiahao Ruan, Junhao Jiao, Kechen Tang, Hongyin Wang, Jingang Tong, Xiao Zhu, Jingbo |
| contents | High-order numerical methods enhance Transformer performance in tasks like NLP and CV, but introduce a performance-efficiency trade-off due to increased computational overhead. Our analysis reveals that conventional efficiency techniques, such as distillation, can be detrimental to the performance of these models, exemplified by PCformer. To explore more optimizable ODE-based Transformer architectures, we propose the Iterative Implicit Euler Transformer (IIET), which simplifies high-order methods using an iterative implicit Euler approach. This simplification not only leads to superior performance but also facilitates model compression compared to PCformer. To enhance inference efficiency, we introduce Iteration Influence-Aware Distillation (IIAD). Through a flexible threshold, IIAD allows users to effectively balance the performance-efficiency trade-off. On lm-evaluation-harness, IIET boosts average accuracy by 2.65% over vanilla Transformers and 0.8% over PCformer. Its efficient variant, E-IIET, significantly cuts inference overhead by 55% while retaining 99.4% of the original task accuracy. Moreover, the most efficient IIET variant achieves an average performance gain exceeding 1.6% over vanilla Transformer with comparable speed. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2509_22463 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | IIET: Efficient Numerical Transformer via Implicit Iterative Euler Method Liu, Xinyu Li, Bei Liu, Jiahao Ruan, Junhao Jiao, Kechen Tang, Hongyin Wang, Jingang Tong, Xiao Zhu, Jingbo Machine Learning Computation and Language High-order numerical methods enhance Transformer performance in tasks like NLP and CV, but introduce a performance-efficiency trade-off due to increased computational overhead. Our analysis reveals that conventional efficiency techniques, such as distillation, can be detrimental to the performance of these models, exemplified by PCformer. To explore more optimizable ODE-based Transformer architectures, we propose the Iterative Implicit Euler Transformer (IIET), which simplifies high-order methods using an iterative implicit Euler approach. This simplification not only leads to superior performance but also facilitates model compression compared to PCformer. To enhance inference efficiency, we introduce Iteration Influence-Aware Distillation (IIAD). Through a flexible threshold, IIAD allows users to effectively balance the performance-efficiency trade-off. On lm-evaluation-harness, IIET boosts average accuracy by 2.65% over vanilla Transformers and 0.8% over PCformer. Its efficient variant, E-IIET, significantly cuts inference overhead by 55% while retaining 99.4% of the original task accuracy. Moreover, the most efficient IIET variant achieves an average performance gain exceeding 1.6% over vanilla Transformer with comparable speed. |
| title | IIET: Efficient Numerical Transformer via Implicit Iterative Euler Method |
| topic | Machine Learning Computation and Language |
| url | https://arxiv.org/abs/2509.22463 |