Cover Image

What Kinds of Tokens Benefit from Distant Text? An Analysis on Long Context Language Modeling

Fuente: arXiv

Saved in:

Bibliographic Details
Main Authors:	Hu, Yutong, Huang, Quzhe, Luo, Kangcheng, Feng, Yansong
Format:	Preprint
Published:	2024
Subjects:	Computation and Language
Online Access:	Acceder al recurso
Tags:	Add Tag No Tags, Be the first to tag this record!

Similar Items

Can Perplexity Reflect Large Language Model's Ability in Long Text Understanding?
by: Hu, Yutong, et al.
Published: (2024)

Only One Relation Possible? Modeling the Ambiguity in Event Temporal Relation Extraction
by: Hu, Yutong, et al.
Published: (2024)

Automating Legal Interpretation with LLMs: Retrieval, Generation, and Evaluation
by: Luo, Kangcheng, et al.
Published: (2025)

ELLA: Empowering LLMs for Interpretable, Accurate and Informative Legal Advice
by: Hu, Yutong, et al.
Published: (2024)

D$^2$Plan: Dual-Agent Dynamic Global Planning for Complex Retrieval-Augmented Reasoning
by: Luo, Kangcheng, et al.
Published: (2026)

Probing Multimodal Large Language Models for Global and Local Semantic Representations
by: Tao, Mingxu, et al.
Published: (2024)

MC$^2$: Towards Transparent and Culturally-Aware NLP for Minority Languages in China
by: Zhang, Chen, et al.
Published: (2023)

Unlocking the Potential of Model Merging for Low-Resource Languages
by: Tao, Mingxu, et al.
Published: (2024)

What Kind of Language is Easy to Language-Model Under Curriculum Learning?
by: El-Naggar, Nadine, et al.
Published: (2026)

JUREX-4E: Juridical Expert-Annotated Four-Element Knowledge Base for Legal Reasoning
by: Liu, Huanghai, et al.
Published: (2025)

ProTrix: Building Models for Planning and Reasoning over Tables with Sentence Context
by: Wu, Zirui, et al.
Published: (2024)

SEGMENT+: Long Text Processing with Short-Context Language Models
by: Shi, Wei, et al.
Published: (2024)

SemToken: Semantic-Aware Tokenization for Efficient Long-Context Language Modeling
by: Liu, Dong, et al.
Published: (2025)

RULER: What's the Real Context Size of Your Long-Context Language Models?
by: Hsieh, Cheng-Ping, et al.
Published: (2024)

Harder Tasks Need More Experts: Dynamic Routing in MoE Models
by: Huang, Quzhe, et al.
Published: (2024)

No Tokens Wasted: Leveraging Long Context in Biomedical Vision-Language Models
by: Sun, Min Woo, et al.
Published: (2025)

PACE: Improving Prompt with Actor-Critic Editing for Large Language Model
by: Dong, Yihong, et al.
Published: (2023)

Every Token Counts: Generalizing 16M Ultra-Long Context in Large Language Models
by: Hu, Xiang, et al.
Published: (2025)

LoPT: Lossless Parallel Tokenization Acceleration for Long Context Inference of Large Language Model
by: Shao, Wei, et al.
Published: (2025)

Training Text-to-Molecule Models with Context-Aware Tokenization
by: Kim, Seojin, et al.
Published: (2025)

Teaching Large Language Models an Unseen Language on the Fly
by: Zhang, Chen, et al.
Published: (2024)

HieraVid: Hierarchical Token Pruning for Fast Video Large Language Models
by: Guo, Yansong, et al.
Published: (2026)

Long Context is Not Long at All: A Prospector of Long-Dependency Data for Large Language Models
by: Chen, Longze, et al.
Published: (2024)

Token Weighting for Long-Range Language Modeling
by: Helm, Falko, et al.
Published: (2025)

On Linearizing Structured Data in Encoder-Decoder Language Models: Insights from Text-to-SQL
by: Shao, Yutong, et al.
Published: (2024)

Text-to-Distribution Prediction with Quantile Tokens and Neighbor Context
by: Zhu, Yilun, et al.
Published: (2026)

Decoupling the Benefits of Subword Tokenization for Language Model Training via Byte-level Simulation
by: Gigant, Théo, et al.
Published: (2026)

Locality Matters for Training-Free Audio Token Compression in Audio-Language Models
by: Luo, Jiale, et al.
Published: (2026)

Thus Spake Long-Context Large Language Model
by: Liu, Xiaoran, et al.
Published: (2025)

A Controllable Examination for Long-Context Language Models
by: Yang, Yijun, et al.
Published: (2025)

Combining Distantly Supervised Models with In Context Learning for Monolingual and Cross-Lingual Relation Extraction
by: Rathore, Vipul, et al.
Published: (2025)

Long-Context Language Modeling with Parallel Context Encoding
by: Yen, Howard, et al.
Published: (2024)

What Language is This? Ask Your Tokenizer
by: Meister, Clara, et al.
Published: (2026)

Exploiting Sparsity for Long Context Inference: Million Token Contexts on Commodity GPUs
by: Synk, Ryan, et al.
Published: (2025)

LongRecipe: Recipe for Efficient Long Context Generalization in Large Language Models
by: Hu, Zhiyuan, et al.
Published: (2024)

Video-LaVIT: Unified Video-Language Pre-training with Decoupled Visual-Motional Tokenization
by: Jin, Yang, et al.
Published: (2024)

$\infty$Bench: Extending Long Context Evaluation Beyond 100K Tokens
by: Zhang, Xinrong, et al.
Published: (2024)

Training-Free Long-Context Scaling of Large Language Models
by: An, Chenxin, et al.
Published: (2024)

SlimInfer: Accelerating Long-Context LLM Inference via Dynamic Token Pruning
by: Long, Lingkun, et al.
Published: (2025)

DAM: Dynamic Attention Mask for Long-Context Large Language Model Inference Acceleration
by: Zhang, Hanzhi, et al.
Published: (2025)