Saved in:
| Main Authors: | , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2412.13626 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866910750776229888 |
|---|---|
| author | Mao, Yansheng Li, Jiaqi Meng, Fanxu Xiong, Jing Zheng, Zilong Zhang, Muhan |
| author_facet | Mao, Yansheng Li, Jiaqi Meng, Fanxu Xiong, Jing Zheng, Zilong Zhang, Muhan |
| contents | Long context understanding remains challenging for large language models due to their limited context windows. This paper introduces Long Input Fine-Tuning (LIFT) for long context modeling, a novel framework that enhances LLM performance on long-context tasks by adapting model parameters to the context at test time. LIFT enables efficient processing of lengthy inputs without the computational burden of offline long-context adaptation, and can improve the long-context capabilities of arbitrary short-context models. The framework is further enhanced by integrating in-context learning and pre-LIFT supervised fine-tuning. The combination of in-context learning and LIFT enables short-context models like Llama 3 to handle arbitrarily long contexts and consistently improves their performance on popular long-context benchmarks like LooGLE and LongBench. We also provide a comprehensive analysis of the strengths and limitations of LIFT on long context understanding, offering valuable directions for future research. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2412_13626 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | LIFT: Improving Long Context Understanding Through Long Input Fine-Tuning Mao, Yansheng Li, Jiaqi Meng, Fanxu Xiong, Jing Zheng, Zilong Zhang, Muhan Computation and Language Artificial Intelligence Long context understanding remains challenging for large language models due to their limited context windows. This paper introduces Long Input Fine-Tuning (LIFT) for long context modeling, a novel framework that enhances LLM performance on long-context tasks by adapting model parameters to the context at test time. LIFT enables efficient processing of lengthy inputs without the computational burden of offline long-context adaptation, and can improve the long-context capabilities of arbitrary short-context models. The framework is further enhanced by integrating in-context learning and pre-LIFT supervised fine-tuning. The combination of in-context learning and LIFT enables short-context models like Llama 3 to handle arbitrarily long contexts and consistently improves their performance on popular long-context benchmarks like LooGLE and LongBench. We also provide a comprehensive analysis of the strengths and limitations of LIFT on long context understanding, offering valuable directions for future research. |
| title | LIFT: Improving Long Context Understanding Through Long Input Fine-Tuning |
| topic | Computation and Language Artificial Intelligence |
| url | https://arxiv.org/abs/2412.13626 |