RouteLMT: Learned Sample Routing for Hybrid LLM Translation Deployment
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2026
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866915955240599552 |
|---|---|
| author | Luo, Yingfeng Liu, Hongyu Lin, Dingyang Chang, Kaiyan Wang, Chenglong Li, Bei Du, Quan Xiao, Tong Zhu, Jingbo |
| author_facet | Luo, Yingfeng Liu, Hongyu Lin, Dingyang Chang, Kaiyan Wang, Chenglong Li, Bei Du, Quan Xiao, Tong Zhu, Jingbo |
| contents | Large Language Models (LLMs) have achieved remarkable performance in Machine Translation (MT), but deploying them at scale remains prohibitively expensive. A widely adopted remedy is the hybrid system paradigm, which balances cost and quality by serving most requests with a small model and selectively routing a fraction to a large model. However, existing routing strategies often rely on heuristics, external predictors, or absolute quality estimation, which fail to capture whether the large model actually provides a worthwhile improvement over the small one. In this paper, we formulate routing as a budget allocation problem and identify marginal gain, i.e., the large model's improvement over the small model, as the optimal signal for budgeted decisions. Building on this, we propose \textbf{RouteLMT} (routing for LLM-based MT), an efficient in-model router that predicts this expected gain by probing the small translators prompt-token representation, without requiring external models or hypothesis decoding. Extensive experiments demonstrate that our RouteLMT outperforms heuristics, quality/difficulty estimation baselines, achieving a superior quality-budget Pareto frontier. Furthermore, we analyze regression risks and show that a simple guarded variant can mitigate severe quality losses. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2604_22520 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | RouteLMT: Learned Sample Routing for Hybrid LLM Translation Deployment Luo, Yingfeng Liu, Hongyu Lin, Dingyang Chang, Kaiyan Wang, Chenglong Li, Bei Du, Quan Xiao, Tong Zhu, Jingbo Computation and Language Large Language Models (LLMs) have achieved remarkable performance in Machine Translation (MT), but deploying them at scale remains prohibitively expensive. A widely adopted remedy is the hybrid system paradigm, which balances cost and quality by serving most requests with a small model and selectively routing a fraction to a large model. However, existing routing strategies often rely on heuristics, external predictors, or absolute quality estimation, which fail to capture whether the large model actually provides a worthwhile improvement over the small one. In this paper, we formulate routing as a budget allocation problem and identify marginal gain, i.e., the large model's improvement over the small model, as the optimal signal for budgeted decisions. Building on this, we propose \textbf{RouteLMT} (routing for LLM-based MT), an efficient in-model router that predicts this expected gain by probing the small translators prompt-token representation, without requiring external models or hypothesis decoding. Extensive experiments demonstrate that our RouteLMT outperforms heuristics, quality/difficulty estimation baselines, achieving a superior quality-budget Pareto frontier. Furthermore, we analyze regression risks and show that a simple guarded variant can mitigate severe quality losses. |
| title | RouteLMT: Learned Sample Routing for Hybrid LLM Translation Deployment |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2604.22520 |