Image de couverture de livre

Scaling Context, Not Parameters: Training a Compact 7B Language Model for Efficient Long-Context Processing

Fuente: arXiv

Enregistré dans:

Détails bibliographiques
Auteurs principaux:	Wu, Chen, Song, Yin
Format:	Preprint
Publié:	2025
Sujets:	Computation and Language Machine Learning
Accès en ligne:	Acceder al recurso
Tags:	Ajouter un tag Pas de tags, Soyez le premier à ajouter un tag!

Documents similaires

How to Train Long-Context Language Models (Effectively)
par: Gao, Tianyu, et autres
Publié: (2024)

PERK: Long-Context Reasoning as Parameter-Efficient Test-Time Learning
par: Chen, Zeming, et autres
Publié: (2025)

HMT: Hierarchical Memory Transformer for Efficient Long Context Language Processing
par: He, Zifan, et autres
Publié: (2024)

Core Context Aware Transformers for Long Context Language Modeling
par: Chen, Yaofo, et autres
Publié: (2024)

LongLoRA: Efficient Fine-tuning of Long-Context Large Language Models
par: Chen, Yukang, et autres
Publié: (2023)

Latent Context Compilation: Distilling Long Context into Compact Portable Memory
par: Li, Zeju, et autres
Publié: (2026)

A Comprehensive Survey on Long Context Language Modeling
par: Liu, Jiaheng, et autres
Publié: (2025)

LongEmbed: Extending Embedding Models for Long Context Retrieval
par: Zhu, Dawei, et autres
Publié: (2024)

Revisiting In-Context Learning with Long Context Language Models
par: Baek, Jinheon, et autres
Publié: (2024)

InfLLM: Training-Free Long-Context Extrapolation for LLMs with an Efficient Context Memory
par: Xiao, Chaojun, et autres
Publié: (2024)

Compress, Gather, and Recompute: REFORMing Long-Context Processing in Transformers
par: Song, Woomin, et autres
Publié: (2025)

From 128K to 4M: Efficient Training of Ultra-Long Context Large Language Models
par: Xu, Chejian, et autres
Publié: (2025)

Long Context RAG Performance of Large Language Models
par: Leng, Quinn, et autres
Publié: (2024)

Let's (not) just put things in Context: Test-Time Training for Long-Context LLMs
par: Bansal, Rachit, et autres
Publié: (2025)

Scaling Long-Horizon LLM Agent via Context-Folding
par: Sun, Weiwei, et autres
Publié: (2025)

Quest: Query-Aware Sparsity for Efficient Long-Context LLM Inference
par: Tang, Jiaming, et autres
Publié: (2024)

Multipole Attention for Efficient Long Context Reasoning
par: Hooper, Coleman, et autres
Publié: (2025)

Token Sparse Attention: Efficient Long-Context Inference with Interleaved Token Selection
par: Jo, Dongwon, et autres
Publié: (2026)

LongAlign: A Recipe for Long Context Alignment of Large Language Models
par: Bai, Yushi, et autres
Publié: (2024)

LongPO: Long Context Self-Evolution of Large Language Models through Short-to-Long Preference Optimization
par: Chen, Guanzheng, et autres
Publié: (2025)

Probe and Skip: Self-Predictive Token Skipping for Efficient Long-Context LLM Inference
par: Wu, Zimeng, et autres
Publié: (2026)

PoSE: Efficient Context Window Extension of LLMs via Positional Skip-wise Training
par: Zhu, Dawei, et autres
Publié: (2023)

Efficient Training of Language Models with Compact and Consistent Next Token Distributions
par: Sathe, Ashutosh, et autres
Publié: (2024)

Training-Inference Consistent Segmented Execution for Long-Context LLMs
par: Shang, Xianpeng, et autres
Publié: (2026)

Samba: Simple Hybrid State Space Models for Efficient Unlimited Context Language Modeling
par: Ren, Liliang, et autres
Publié: (2024)

Found in the Middle: How Language Models Use Long Contexts Better via Plug-and-Play Positional Encoding
par: Zhang, Zhenyu, et autres
Publié: (2024)

Cost-Optimal Grouped-Query Attention for Long-Context Modeling
par: Chen, Yingfa, et autres
Publié: (2025)

Artificial Hippocampus Networks for Efficient Long-Context Modeling
par: Fang, Yunhao, et autres
Publié: (2025)

Extending Input Contexts of Language Models through Training on Segmented Sequences
par: Karypis, Petros, et autres
Publié: (2023)

CSKV: Training-Efficient Channel Shrinking for KV Cache in Long-Context Scenarios
par: Wang, Luning, et autres
Publié: (2024)

Systematic Evaluation of Optimization Techniques for Long-Context Language Models
par: Ahmed, Ammar, et autres
Publié: (2025)

Compressed Context Memory For Online Language Model Interaction
par: Kim, Jang-Hyun, et autres
Publié: (2023)

Retrieval meets Long Context Large Language Models
par: Xu, Peng, et autres
Publié: (2023)

MOM: Memory-Efficient Offloaded Mini-Sequence Inference for Long Context Language Models
par: Zhang, Junyang, et autres
Publié: (2025)

NeuroLoRA: Context-Aware Neuromodulation for Parameter-Efficient Multi-Task Adaptation
par: Yang, Yuxin, et autres
Publié: (2026)

Long-Short Alignment for Effective Long-Context Modeling in LLMs
par: Du, Tianqi, et autres
Publié: (2025)

MTraining: Distributed Dynamic Sparse Attention for Efficient Ultra-Long Context Training
par: Li, Wenxuan, et autres
Publié: (2025)

LongReD: Mitigating Short-Text Degradation of Long-Context Large Language Models via Restoration Distillation
par: Dong, Zican, et autres
Publié: (2025)

Efficient Context Propagating Perceiver Architectures for Auto-Regressive Language Modeling
par: Mahmood, Kaleel, et autres
Publié: (2024)

100-LongBench: Are de facto Long-Context Benchmarks Literally Evaluating Long-Context Ability?
par: Yang, Wang, et autres
Publié: (2025)