Cover Image

Every Token Counts: Generalizing 16M Ultra-Long Context in Large Language Models

Fuente: arXiv

Saved in:

Bibliographic Details
Main Authors:	Hu, Xiang, Zhou, Zhanchao, Liang, Ruiqi, Li, Zehuan, Wu, Wei, Li, Jianguo
Format:	Preprint
Published:	2025
Subjects:	Computation and Language Artificial Intelligence
Online Access:	Acceder al recurso
Tags:	Add Tag No Tags, Be the first to tag this record!

Similar Items

Counting Ability of Large Language Models and Impact of Tokenization
by: Zhang, Xiang, et al.
Published: (2024)

When Every Token Counts: Optimal Segmentation for Low-Resource Language Models
by: Raj, Bharath, et al.
Published: (2024)

From 128K to 4M: Efficient Training of Ultra-Long Context Large Language Models
by: Xu, Chejian, et al.
Published: (2025)

Do We Need Distinct Representations for Every Speech Token? Unveiling and Exploiting Redundancy in Large Speech Language Models
by: Xiang, Bajian, et al.
Published: (2026)

Groma: Localized Visual Tokenization for Grounding Multimodal Large Language Models
by: Ma, Chuofan, et al.
Published: (2024)

LongSkywork: A Training Recipe for Efficiently Extending Context Length in Large Language Models
by: Zhao, Liang, et al.
Published: (2024)

Efficient Length-Generalizable Attention via Causal Retrieval for Long-Context Language Modeling
by: Hu, Xiang, et al.
Published: (2024)

TIDE: Every Layer Knows the Token Beneath the Context
by: Jaiswal, Ajay, et al.
Published: (2026)

Every Attention Matters: An Efficient Hybrid Architecture for Long-Context Reasoning
by: Ling Team, et al.
Published: (2025)

SemToken: Semantic-Aware Tokenization for Efficient Long-Context Language Modeling
by: Liu, Dong, et al.
Published: (2025)

Understanding and Improving Length Generalization in Hierarchical Sparse Attention Models
by: Leng, Jiaqi, et al.
Published: (2025)

Make Every Penny Count: Difficulty-Adaptive Self-Consistency for Cost-Efficient Reasoning
by: Wang, Xinglin, et al.
Published: (2024)

Exposing Numeracy Gaps: A Benchmark to Evaluate Fundamental Numerical Abilities in Large Language Models
by: Li, Haoyang, et al.
Published: (2025)

Large Language Models are In-Context Molecule Learners
by: Li, Jiatong, et al.
Published: (2024)

Tokenization Matters! Degrading Large Language Models through Challenging Their Tokenization
by: Wang, Dixuan, et al.
Published: (2024)

Inclusion Arena: An Open Platform for Evaluating Large Foundation Models with Real-World Apps
by: Wang, Kangyu, et al.
Published: (2025)

Every Step Counts: Decoding Trajectories as Authorship Fingerprints of dLLMs
by: Li, Qi, et al.
Published: (2025)

LongWriter-Zero: Mastering Ultra-Long Text Generation via Reinforcement Learning
by: Wu, Yuhao, et al.
Published: (2025)

LooGLE: Can Long-Context Language Models Understand Long Contexts?
by: Li, Jiaqi, et al.
Published: (2023)

CNNSum: Exploring Long-Context Summarization with Large Language Models in Chinese Novels
by: Wei, Lingxiao, et al.
Published: (2024)

Retrieval meets Long Context Large Language Models
by: Xu, Peng, et al.
Published: (2023)

LongSafety: Evaluating Long-Context Safety of Large Language Models
by: Lu, Yida, et al.
Published: (2025)

UniMem: Towards a Unified View of Long-Context Large Language Models
by: Fang, Junjie, et al.
Published: (2024)

Diagnosing Retrieval Bias Under Multiple In-Context Knowledge Updates in Large Language Models
by: Qiao, Boyu, et al.
Published: (2026)

Every FLOP Counts: Scaling a 300B Mixture-of-Experts LING LLM without Premium GPUs
by: Ling Team, et al.
Published: (2025)

CoCA: Fusing Position Embedding with Collinear Constrained Attention in Transformers for Long Context Window Extending
by: Zhu, Shiyi, et al.
Published: (2023)

Every Activation Boosted: Scaling General Reasoner to 1 Trillion Open Language Foundation
by: Ling Team, et al.
Published: (2025)

UltraLLaDA: Scaling the Context Length to 128K for Diffusion Large Language Models
by: He, Guangxin, et al.
Published: (2025)

Token Signature: Predicting Chain-of-Thought Gains with Token Decoding Feature in Large Language Models
by: Liu, Peijie, et al.
Published: (2025)

E2LLM: Encoder Elongated Large Language Models for Long-Context Understanding and Reasoning
by: Liao, Zihan, et al.
Published: (2024)

Micro-Macro Retrieval: Reducing Long-Form Hallucination in Large Language Models
by: Feng, Yujie, et al.
Published: (2026)

Large Language Models are Miscalibrated In-Context Learners
by: Li, Chengzu, et al.
Published: (2023)

Counterfactual Token Generation in Large Language Models
by: Chatzi, Ivi, et al.
Published: (2024)

LongWriter-V: Enabling Ultra-Long and High-Fidelity Generation in Vision-Language Models
by: Tu, Shangqing, et al.
Published: (2025)

Every Part Matters: Integrity Verification of Scientific Figures Based on Multimodal Large Language Models
by: Shi, Xiang, et al.
Published: (2024)

GALLa: Graph Aligned Large Language Models for Improved Source Code Understanding
by: Zhang, Ziyin, et al.
Published: (2024)

Unveiling Selection Biases: Exploring Order and Token Sensitivity in Large Language Models
by: Wei, Sheng-Lun, et al.
Published: (2024)

GraphReader: Building Graph-based Agent to Enhance Long-Context Abilities of Large Language Models
by: Li, Shilong, et al.
Published: (2024)

VITA-Audio: Fast Interleaved Cross-Modal Token Generation for Efficient Large Speech-Language Model
by: Long, Zuwei, et al.
Published: (2025)

Memory Retrieval and Consolidation in Large Language Models through Function Tokens
by: Zhang, Shaohua, et al.
Published: (2025)