End-to-End Test-Time Training for Long Context
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866908740078272512 |
|---|---|
| author | Tandon, Arnuv Dalal, Karan Li, Xinhao Koceja, Daniel Rød, Marcel Buchanan, Sam Wang, Xiaolong Leskovec, Jure Koyejo, Sanmi Hashimoto, Tatsunori Guestrin, Carlos McCaleb, Jed Choi, Yejin Sun, Yu |
| author_facet | Tandon, Arnuv Dalal, Karan Li, Xinhao Koceja, Daniel Rød, Marcel Buchanan, Sam Wang, Xiaolong Leskovec, Jure Koyejo, Sanmi Hashimoto, Tatsunori Guestrin, Carlos McCaleb, Jed Choi, Yejin Sun, Yu |
| contents | We formulate long-context language modeling as a problem in continual learning rather than architecture design. Under this formulation, we only use a standard architecture -- a Transformer with sliding-window attention. However, our model continues learning at test time via next-token prediction on the given context, compressing the context it reads into its weights. In addition, we improve the model's initialization for learning at test time via meta-learning at training time. Overall, our method, a form of Test-Time Training (TTT), is End-to-End (E2E) both at test time (via next-token prediction) and training time (via meta-learning), in contrast to previous forms. We conduct extensive experiments with a focus on scaling properties. In particular, for 3B models trained with 164B tokens, our method (TTT-E2E) scales with context length in the same way as Transformer with full attention, while others, such as Mamba 2 and Gated DeltaNet, do not. However, similar to RNNs, TTT-E2E has constant inference latency regardless of context length, making it 2.7 times faster than full attention for 128K context. Our code is publicly available. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2512_23675 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | End-to-End Test-Time Training for Long Context Tandon, Arnuv Dalal, Karan Li, Xinhao Koceja, Daniel Rød, Marcel Buchanan, Sam Wang, Xiaolong Leskovec, Jure Koyejo, Sanmi Hashimoto, Tatsunori Guestrin, Carlos McCaleb, Jed Choi, Yejin Sun, Yu Machine Learning We formulate long-context language modeling as a problem in continual learning rather than architecture design. Under this formulation, we only use a standard architecture -- a Transformer with sliding-window attention. However, our model continues learning at test time via next-token prediction on the given context, compressing the context it reads into its weights. In addition, we improve the model's initialization for learning at test time via meta-learning at training time. Overall, our method, a form of Test-Time Training (TTT), is End-to-End (E2E) both at test time (via next-token prediction) and training time (via meta-learning), in contrast to previous forms. We conduct extensive experiments with a focus on scaling properties. In particular, for 3B models trained with 164B tokens, our method (TTT-E2E) scales with context length in the same way as Transformer with full attention, while others, such as Mamba 2 and Gated DeltaNet, do not. However, similar to RNNs, TTT-E2E has constant inference latency regardless of context length, making it 2.7 times faster than full attention for 128K context. Our code is publicly available. |
| title | End-to-End Test-Time Training for Long Context |
| topic | Machine Learning |
| url | https://arxiv.org/abs/2512.23675 |