Saved in:
| Main Authors: | , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2026
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2605.10544 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866910209242300416 |
|---|---|
| author | Zhu, Jinchang Li, Jindong Zou, Chengyu Fu, Rong Wang, Chao He, Haowei Yang, Menglin |
| author_facet | Zhu, Jinchang Li, Jindong Zou, Chengyu Fu, Rong Wang, Chao He, Haowei Yang, Menglin |
| contents | Long-context adaptation is often viewed as window scaling, but this misses a token-level supervision mismatch: in packed training with document masking, each target token's effective context remains short. We introduce EXACT, a supervision-allocation objective that assigns extra weight to long effective-context targets by inverse frequency within the long tail. Across seven Qwen/LLaMA CPT configurations, EXACT improves all 28 trained/extrapolated NoLiMa and RULER comparisons. On Qwen2.5-0.5B, NoLiMa improves by +10.09 (trained) and +5.34 (extrapolated); RULER by +10.69 and +5.55. On LLaMA-3.2-3B, RULER improves by +17.91 and +16.11. Standard QA/reasoning are preserved (+0.24 macro change across six benchmarks). A distance-resolved probe shows gains arise when evidence is thousands of tokens away, while short cases remain unchanged. Results support a supervision-centric thesis: long-context adaptation depends on how strongly training supervises long-context predictions. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2605_10544 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | Where Does Long-Context Supervision Actually Go? Effective-Context Exposure Balancing Zhu, Jinchang Li, Jindong Zou, Chengyu Fu, Rong Wang, Chao He, Haowei Yang, Menglin Computation and Language Long-context adaptation is often viewed as window scaling, but this misses a token-level supervision mismatch: in packed training with document masking, each target token's effective context remains short. We introduce EXACT, a supervision-allocation objective that assigns extra weight to long effective-context targets by inverse frequency within the long tail. Across seven Qwen/LLaMA CPT configurations, EXACT improves all 28 trained/extrapolated NoLiMa and RULER comparisons. On Qwen2.5-0.5B, NoLiMa improves by +10.09 (trained) and +5.34 (extrapolated); RULER by +10.69 and +5.55. On LLaMA-3.2-3B, RULER improves by +17.91 and +16.11. Standard QA/reasoning are preserved (+0.24 macro change across six benchmarks). A distance-resolved probe shows gains arise when evidence is thousands of tokens away, while short cases remain unchanged. Results support a supervision-centric thesis: long-context adaptation depends on how strongly training supervises long-context predictions. |
| title | Where Does Long-Context Supervision Actually Go? Effective-Context Exposure Balancing |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2605.10544 |