Saved in:
| Main Authors: | , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2026
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2605.28053 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866911723668111360 |
|---|---|
| author | Yang, Jian Kou, Zhizhuo Tian, Yao Zhang, Hao Chen, Han Han, Sirui Guo, Yike |
| author_facet | Yang, Jian Kou, Zhizhuo Tian, Yao Zhang, Hao Chen, Han Han, Sirui Guo, Yike |
| contents | Test-time training (TTT) adapts an LLM during generation by reading and updating request-owned state, such as fast weights, low-rank deltas, or streaming learner state. This breaks batched LLM serving, which assumes shared static weights: serial execution is correct but slow, while naive batching can corrupt request state. We formulate this problem as read-write TTT serving and present RW-TTT , which tags each decode step with its owner, version, and READ/WRITE effect, batches only compatible phases, and commits updates only to the owner. On one GPU with eight fast-weight InPlace-TTT streams, RW-TTT reaches 274.61 aggregate tok/s, 9.31x over sequential serving and 3.44x over per-stream replicas under the same memory budget. It preserves behavior on RULER, a long-context benchmark, and passes owner/version checks. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2605_28053 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | RW-TTT: Batched Serving for Request-Owned Test-Time Training State Yang, Jian Kou, Zhizhuo Tian, Yao Zhang, Hao Chen, Han Han, Sirui Guo, Yike Machine Learning Test-time training (TTT) adapts an LLM during generation by reading and updating request-owned state, such as fast weights, low-rank deltas, or streaming learner state. This breaks batched LLM serving, which assumes shared static weights: serial execution is correct but slow, while naive batching can corrupt request state. We formulate this problem as read-write TTT serving and present RW-TTT , which tags each decode step with its owner, version, and READ/WRITE effect, batches only compatible phases, and commits updates only to the owner. On one GPU with eight fast-weight InPlace-TTT streams, RW-TTT reaches 274.61 aggregate tok/s, 9.31x over sequential serving and 3.44x over per-stream replicas under the same memory budget. It preserves behavior on RULER, a long-context benchmark, and passes owner/version checks. |
| title | RW-TTT: Batched Serving for Request-Owned Test-Time Training State |
| topic | Machine Learning |
| url | https://arxiv.org/abs/2605.28053 |