SELF: Self-Extend the Context Length With Logistic Growth Function
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866915300303175680 |
|---|---|
| author | Dang, Phat Thanh Thoppay, Saahil Yang, Wang Wang, Qifan Chaudhary, Vipin Han, Xiaotian |
| author_facet | Dang, Phat Thanh Thoppay, Saahil Yang, Wang Wang, Qifan Chaudhary, Vipin Han, Xiaotian |
| contents | Large language models suffer issues when operated on long contexts that are larger than their training context length due to the standard position encoding for tokens in the attention layer. Tokens a long distance apart will rarely have an effect on each other and long prompts yield unexpected results. To solve this problem, we propose SELF (Self-Extend the Context Length With Logistic Growth Function): a solution of grouping consecutive tokens at varying group sizes using a logistic capacity equation combined with a constant group size at smaller relative distances. Our model had an increase in performance of up to 12% compared to the LongLM extension method in LEval (specifically on the Qwen model). On summarization related tasks in LongBench, our model performed up to 6.4% better than LongLM (specifically on the Llama-2-7b model). On reading comprehension tasks from LEval, our model performed up to 5.4% better than the LongLM. Our code is available at https://github.com/alexeipc/SELF-LLM. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2505_17296 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | SELF: Self-Extend the Context Length With Logistic Growth Function Dang, Phat Thanh Thoppay, Saahil Yang, Wang Wang, Qifan Chaudhary, Vipin Han, Xiaotian Computation and Language Machine Learning Large language models suffer issues when operated on long contexts that are larger than their training context length due to the standard position encoding for tokens in the attention layer. Tokens a long distance apart will rarely have an effect on each other and long prompts yield unexpected results. To solve this problem, we propose SELF (Self-Extend the Context Length With Logistic Growth Function): a solution of grouping consecutive tokens at varying group sizes using a logistic capacity equation combined with a constant group size at smaller relative distances. Our model had an increase in performance of up to 12% compared to the LongLM extension method in LEval (specifically on the Qwen model). On summarization related tasks in LongBench, our model performed up to 6.4% better than LongLM (specifically on the Llama-2-7b model). On reading comprehension tasks from LEval, our model performed up to 5.4% better than the LongLM. Our code is available at https://github.com/alexeipc/SELF-LLM. |
| title | SELF: Self-Extend the Context Length With Logistic Growth Function |
| topic | Computation and Language Machine Learning |
| url | https://arxiv.org/abs/2505.17296 |