Imagen de Portada

Scaling Context, Not Parameters: Training a Compact 7B Language Model for Efficient Long-Context Processing

Fuente: arXiv

Guardado en:

Detalles Bibliográficos
Autores principales:	Wu, Chen, Song, Yin
Formato:	Preprint
Publicado:	2025
Materias:	Computation and Language Machine Learning
Acceso en línea:	Acceder al recurso
Etiquetas:	Agregar Etiqueta Sin Etiquetas, Sea el primero en etiquetar este registro!

Ejemplares similares

How to Train Long-Context Language Models (Effectively)
por: Gao, Tianyu, et al.
Publicado: (2024)

PERK: Long-Context Reasoning as Parameter-Efficient Test-Time Learning
por: Chen, Zeming, et al.
Publicado: (2025)

HMT: Hierarchical Memory Transformer for Efficient Long Context Language Processing
por: He, Zifan, et al.
Publicado: (2024)

Core Context Aware Transformers for Long Context Language Modeling
por: Chen, Yaofo, et al.
Publicado: (2024)

LongLoRA: Efficient Fine-tuning of Long-Context Large Language Models
por: Chen, Yukang, et al.
Publicado: (2023)

Latent Context Compilation: Distilling Long Context into Compact Portable Memory
por: Li, Zeju, et al.
Publicado: (2026)

A Comprehensive Survey on Long Context Language Modeling
por: Liu, Jiaheng, et al.
Publicado: (2025)

LongEmbed: Extending Embedding Models for Long Context Retrieval
por: Zhu, Dawei, et al.
Publicado: (2024)

Revisiting In-Context Learning with Long Context Language Models
por: Baek, Jinheon, et al.
Publicado: (2024)

InfLLM: Training-Free Long-Context Extrapolation for LLMs with an Efficient Context Memory
por: Xiao, Chaojun, et al.
Publicado: (2024)

Compress, Gather, and Recompute: REFORMing Long-Context Processing in Transformers
por: Song, Woomin, et al.
Publicado: (2025)

From 128K to 4M: Efficient Training of Ultra-Long Context Large Language Models
por: Xu, Chejian, et al.
Publicado: (2025)

Long Context RAG Performance of Large Language Models
por: Leng, Quinn, et al.
Publicado: (2024)

Let's (not) just put things in Context: Test-Time Training for Long-Context LLMs
por: Bansal, Rachit, et al.
Publicado: (2025)

Scaling Long-Horizon LLM Agent via Context-Folding
por: Sun, Weiwei, et al.
Publicado: (2025)

Quest: Query-Aware Sparsity for Efficient Long-Context LLM Inference
por: Tang, Jiaming, et al.
Publicado: (2024)

Multipole Attention for Efficient Long Context Reasoning
por: Hooper, Coleman, et al.
Publicado: (2025)

Token Sparse Attention: Efficient Long-Context Inference with Interleaved Token Selection
por: Jo, Dongwon, et al.
Publicado: (2026)

LongAlign: A Recipe for Long Context Alignment of Large Language Models
por: Bai, Yushi, et al.
Publicado: (2024)

LongPO: Long Context Self-Evolution of Large Language Models through Short-to-Long Preference Optimization
por: Chen, Guanzheng, et al.
Publicado: (2025)

Probe and Skip: Self-Predictive Token Skipping for Efficient Long-Context LLM Inference
por: Wu, Zimeng, et al.
Publicado: (2026)

PoSE: Efficient Context Window Extension of LLMs via Positional Skip-wise Training
por: Zhu, Dawei, et al.
Publicado: (2023)

Efficient Training of Language Models with Compact and Consistent Next Token Distributions
por: Sathe, Ashutosh, et al.
Publicado: (2024)

Training-Inference Consistent Segmented Execution for Long-Context LLMs
por: Shang, Xianpeng, et al.
Publicado: (2026)

Samba: Simple Hybrid State Space Models for Efficient Unlimited Context Language Modeling
por: Ren, Liliang, et al.
Publicado: (2024)

Found in the Middle: How Language Models Use Long Contexts Better via Plug-and-Play Positional Encoding
por: Zhang, Zhenyu, et al.
Publicado: (2024)

Cost-Optimal Grouped-Query Attention for Long-Context Modeling
por: Chen, Yingfa, et al.
Publicado: (2025)

Artificial Hippocampus Networks for Efficient Long-Context Modeling
por: Fang, Yunhao, et al.
Publicado: (2025)

Extending Input Contexts of Language Models through Training on Segmented Sequences
por: Karypis, Petros, et al.
Publicado: (2023)

CSKV: Training-Efficient Channel Shrinking for KV Cache in Long-Context Scenarios
por: Wang, Luning, et al.
Publicado: (2024)

Systematic Evaluation of Optimization Techniques for Long-Context Language Models
por: Ahmed, Ammar, et al.
Publicado: (2025)

Compressed Context Memory For Online Language Model Interaction
por: Kim, Jang-Hyun, et al.
Publicado: (2023)

Retrieval meets Long Context Large Language Models
por: Xu, Peng, et al.
Publicado: (2023)

MOM: Memory-Efficient Offloaded Mini-Sequence Inference for Long Context Language Models
por: Zhang, Junyang, et al.
Publicado: (2025)

NeuroLoRA: Context-Aware Neuromodulation for Parameter-Efficient Multi-Task Adaptation
por: Yang, Yuxin, et al.
Publicado: (2026)

Long-Short Alignment for Effective Long-Context Modeling in LLMs
por: Du, Tianqi, et al.
Publicado: (2025)

MTraining: Distributed Dynamic Sparse Attention for Efficient Ultra-Long Context Training
por: Li, Wenxuan, et al.
Publicado: (2025)

LongReD: Mitigating Short-Text Degradation of Long-Context Large Language Models via Restoration Distillation
por: Dong, Zican, et al.
Publicado: (2025)

Efficient Context Propagating Perceiver Architectures for Auto-Regressive Language Modeling
por: Mahmood, Kaleel, et al.
Publicado: (2024)

100-LongBench: Are de facto Long-Context Benchmarks Literally Evaluating Long-Context Ability?
por: Yang, Wang, et al.
Publicado: (2025)