Cover Image

Long-Context Language Modeling with Parallel Context Encoding

Fuente: arXiv

Saved in:

Bibliographic Details
Main Authors:	Yen, Howard, Gao, Tianyu, Chen, Danqi
Format:	Preprint
Published:	2024
Subjects:	Computation and Language
Online Access:	Acceder al recurso
Tags:	Add Tag No Tags, Be the first to tag this record!

Similar Items

How to Train Long-Context Language Models (Effectively)
by: Gao, Tianyu, et al.
Published: (2024)

LongProc: Benchmarking Long-Context Language Models on Long Procedural Generation
by: Ye, Xi, et al.
Published: (2025)

HELMET: How to Evaluate Long-Context Language Models Effectively and Thoroughly
by: Yen, Howard, et al.
Published: (2024)

DySCO: Dynamic Attention-Scaling Decoding for Long-Context Language Models
by: Ye, Xi, et al.
Published: (2026)

Agentic Aggregation for Parallel Scaling of Long-Horizon Agentic Tasks
by: Lee, Yoonsang, et al.
Published: (2026)

Query-Focused Retrieval Heads Improve Long-Context Reasoning and Re-ranking
by: Zhang, Wuwei, et al.
Published: (2025)

Lost in the Maze: Overcoming Context Limitations in Long-Horizon Agentic Search
by: Yen, Howard, et al.
Published: (2025)

Cache Me If You Can: How Many KVs Do You Need for Effective Long-Context LMs?
by: Bhaskar, Adithya, et al.
Published: (2025)

Sheared LLaMA: Accelerating Language Model Pre-training via Structured Pruning
by: Xia, Mengzhou, et al.
Published: (2023)

ParallelComp: Parallel Long-Context Compressor for Length Extrapolation
by: Xiong, Jing, et al.
Published: (2025)

Continual Memorization of Factoids in Language Models
by: Chen, Howard, et al.
Published: (2024)

Improving Language Understanding from Screenshots
by: Gao, Tianyu, et al.
Published: (2024)

Metadata Conditioning Accelerates Language Model Pre-training
by: Gao, Tianyu, et al.
Published: (2025)

Layer-Specific Scaling of Positional Encodings for Superior Long-Context Modeling
by: Wang, Zhenghua, et al.
Published: (2025)

Core Context Aware Transformers for Long Context Language Modeling
by: Chen, Yaofo, et al.
Published: (2024)

LoPT: Lossless Parallel Tokenization Acceleration for Long Context Inference of Large Language Model
by: Shao, Wei, et al.
Published: (2025)

Efficiently Computing Susceptibility to Context in Language Models
by: Liu, Tianyu, et al.
Published: (2024)

Found in the Middle: How Language Models Use Long Contexts Better via Plug-and-Play Positional Encoding
by: Zhang, Zhenyu, et al.
Published: (2024)

Attention Entropy is a Key Factor: An Analysis of Parallel Context Encoding with Full-attention-based Pre-trained Language Models
by: Zhang, Zhisong, et al.
Published: (2024)

Evaluating Large Language Models at Evaluating Instruction Following
by: Zeng, Zhiyuan, et al.
Published: (2023)

LooGLE: Can Long-Context Language Models Understand Long Contexts?
by: Li, Jiaqi, et al.
Published: (2023)

Beyond In-Context Learning: Aligning Long-form Generation of Large Language Models via Task-Inherent Attribute Guidelines
by: Long, Do Xuan, et al.
Published: (2025)

RULER: What's the Real Context Size of Your Long-Context Language Models?
by: Hsieh, Cheng-Ping, et al.
Published: (2024)

A Comprehensive Survey on Long Context Language Modeling
by: Liu, Jiaheng, et al.
Published: (2025)

Revisiting In-Context Learning with Long Context Language Models
by: Baek, Jinheon, et al.
Published: (2024)

Reducing Distraction in Long-Context Language Models by Focused Learning
by: Wu, Zijun, et al.
Published: (2024)

SEGMENT+: Long Text Processing with Short-Context Language Models
by: Shi, Wei, et al.
Published: (2024)

In-Context Learning with Long-Context Models: An In-Depth Exploration
by: Bertsch, Amanda, et al.
Published: (2024)

An Efficient Recipe for Long Context Extension via Middle-Focused Positional Encoding
by: Wu, Tong, et al.
Published: (2024)

In Defense of RAG in the Era of Long-Context Language Models
by: Yu, Tan, et al.
Published: (2024)

Long Context Automated Essay Scoring with Language Models
by: Ormerod, Christopher, et al.
Published: (2025)

A Controllable Examination for Long-Context Language Models
by: Yang, Yijun, et al.
Published: (2025)

Thus Spake Long-Context Large Language Model
by: Liu, Xiaoran, et al.
Published: (2025)

Long-Context Encoder Models for Polish Language Understanding
by: Dadas, Sławomir, et al.
Published: (2026)

Long Context is Not Long at All: A Prospector of Long-Dependency Data for Large Language Models
by: Chen, Longze, et al.
Published: (2024)

Membership Inference Attack against Long-Context Large Language Models
by: Wang, Zixiong, et al.
Published: (2024)

Training-free Context-adaptive Attention for Efficient Long Context Modeling
by: You, Zeng, et al.
Published: (2025)

Language Models that Think, Chat Better
by: Bhaskar, Adithya, et al.
Published: (2025)

Accumulating Context Changes the Beliefs of Language Models
by: Geng, Jiayi, et al.
Published: (2025)

FltLM: An Intergrated Long-Context Large Language Model for Effective Context Filtering and Understanding
by: Deng, Jingyang, et al.
Published: (2024)