Advancing Sequential Numerical Prediction in Autoregressive Models
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866916763878293504 |
|---|---|
| author | Fei, Xiang Lu, Jinghui Sun, Qi Feng, Hao Wang, Yanjie Shi, Wei Wang, An-Lan Tang, Jingqun Huang, Can |
| author_facet | Fei, Xiang Lu, Jinghui Sun, Qi Feng, Hao Wang, Yanjie Shi, Wei Wang, An-Lan Tang, Jingqun Huang, Can |
| contents | Autoregressive models have become the de facto choice for sequence generation tasks, but standard approaches treat digits as independent tokens and apply cross-entropy loss, overlooking the coherent structure of numerical sequences. This paper introduces Numerical Token Integrity Loss (NTIL) to address this gap. NTIL operates at two levels: (1) token-level, where it extends the Earth Mover's Distance (EMD) to preserve ordinal relationships between numerical values, and (2) sequence-level, where it penalizes the overall discrepancy between the predicted and actual sequences. This dual approach improves numerical prediction and integrates effectively with LLMs/MLLMs. Extensive experiments show significant performance improvements with NTIL. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2505_13077 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Advancing Sequential Numerical Prediction in Autoregressive Models Fei, Xiang Lu, Jinghui Sun, Qi Feng, Hao Wang, Yanjie Shi, Wei Wang, An-Lan Tang, Jingqun Huang, Can Computation and Language Artificial Intelligence Machine Learning Autoregressive models have become the de facto choice for sequence generation tasks, but standard approaches treat digits as independent tokens and apply cross-entropy loss, overlooking the coherent structure of numerical sequences. This paper introduces Numerical Token Integrity Loss (NTIL) to address this gap. NTIL operates at two levels: (1) token-level, where it extends the Earth Mover's Distance (EMD) to preserve ordinal relationships between numerical values, and (2) sequence-level, where it penalizes the overall discrepancy between the predicted and actual sequences. This dual approach improves numerical prediction and integrates effectively with LLMs/MLLMs. Extensive experiments show significant performance improvements with NTIL. |
| title | Advancing Sequential Numerical Prediction in Autoregressive Models |
| topic | Computation and Language Artificial Intelligence Machine Learning |
| url | https://arxiv.org/abs/2505.13077 |