Saved in:
| Main Authors: | , , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2508.00370 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866916882987089920 |
|---|---|
| author | Chen, Jiyu Lim, Poh Seng Peng, Shuang Luo, Daxiong Foo, JungHau Deep, Yap Jie, Timothy Lee Jun Wen, Kelvin Teh Kae Yang, Fan Feng, Danyu Chen, Hao-Yun Chen, Peng-Wen Li, Fangyuan Chen, Xiaoxin Mun, Wong Wai |
| author_facet | Chen, Jiyu Lim, Poh Seng Peng, Shuang Luo, Daxiong Foo, JungHau Deep, Yap Jie, Timothy Lee Jun Wen, Kelvin Teh Kae Yang, Fan Feng, Danyu Chen, Hao-Yun Chen, Peng-Wen Li, Fangyuan Chen, Xiaoxin Mun, Wong Wai |
| contents | Deploying Transformer-based large language models (LLMs) on resource-constrained edge devices for long-sequence tasks remains challenging due to the quadratic time complexity of self-attention and growing Key-Value (KV) cache demands. While existing KV cache optimizations improve memory efficiency, they often fail to reduce time to first token (TTFT) and may degrade performance through token pruning. Alternative sequence modeling architectures address some of these limitations, but typically require full retraining and lack infrastructure support. EdgeInfinite offers an efficient solution by fine-tuning only a small subset of parameters, maintaining quality while reducing both computational and memory costs, including improved TTFT. However, its instruction-following ability is limited, and it lacks mobile-specific optimizations. To address these issues, we propose EdgeInfinite-Instruct, which introduces a Segmented Supervised Fine-Tuning (S-SFT) strategy tailored to long-sequence tasks such as summarization and question answering. We further optimized EdgeInfinite-Instruct for efficient deployment on edge NPUs by employing fine-grained post-training quantization (PTQ) to reduce computational demands while maintaining accuracy, and by implementing a fixed-shape computation graph that balances memory usage and on-device efficiency through scenario-specific customization of input token and cache sizes. Experiments on long-context benchmarks and real-world mobile tasks show that our approach improves domain-specific performance while maintaining efficiency on NPU-accelerated edge devices. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2508_00370 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | EdgeInfinite-Instruct: Bridging SFT-Based Optimization and NPU-Level Efficiency for Edge Devices Chen, Jiyu Lim, Poh Seng Peng, Shuang Luo, Daxiong Foo, JungHau Deep, Yap Jie, Timothy Lee Jun Wen, Kelvin Teh Kae Yang, Fan Feng, Danyu Chen, Hao-Yun Chen, Peng-Wen Li, Fangyuan Chen, Xiaoxin Mun, Wong Wai Computation and Language Machine Learning Deploying Transformer-based large language models (LLMs) on resource-constrained edge devices for long-sequence tasks remains challenging due to the quadratic time complexity of self-attention and growing Key-Value (KV) cache demands. While existing KV cache optimizations improve memory efficiency, they often fail to reduce time to first token (TTFT) and may degrade performance through token pruning. Alternative sequence modeling architectures address some of these limitations, but typically require full retraining and lack infrastructure support. EdgeInfinite offers an efficient solution by fine-tuning only a small subset of parameters, maintaining quality while reducing both computational and memory costs, including improved TTFT. However, its instruction-following ability is limited, and it lacks mobile-specific optimizations. To address these issues, we propose EdgeInfinite-Instruct, which introduces a Segmented Supervised Fine-Tuning (S-SFT) strategy tailored to long-sequence tasks such as summarization and question answering. We further optimized EdgeInfinite-Instruct for efficient deployment on edge NPUs by employing fine-grained post-training quantization (PTQ) to reduce computational demands while maintaining accuracy, and by implementing a fixed-shape computation graph that balances memory usage and on-device efficiency through scenario-specific customization of input token and cache sizes. Experiments on long-context benchmarks and real-world mobile tasks show that our approach improves domain-specific performance while maintaining efficiency on NPU-accelerated edge devices. |
| title | EdgeInfinite-Instruct: Bridging SFT-Based Optimization and NPU-Level Efficiency for Edge Devices |
| topic | Computation and Language Machine Learning |
| url | https://arxiv.org/abs/2508.00370 |