Cover Image

LongReD: Mitigating Short-Text Degradation of Long-Context Large Language Models via Restoration Distillation

Fuente: arXiv

Saved in:

Bibliographic Details
Main Authors:	Dong, Zican, Li, Junyi, Jiang, Jinhao, Xu, Mingyu, Zhao, Wayne Xin, Wang, Bingning, Chen, Weipeng
Format:	Preprint
Published:	2025
Subjects:	Computation and Language Machine Learning
Online Access:	Acceder al recurso
Tags:	Add Tag No Tags, Be the first to tag this record!

Similar Items

A Survey on Long Text Modeling with Transformers
by: Dong, Zican, et al.
Published: (2023)

BAMBOO: A Comprehensive Benchmark for Evaluating Long Text Modeling Capacities of Large Language Models
by: Dong, Zican, et al.
Published: (2023)

Exploring Context Window of Large Language Models via Decomposed Positional Vectors
by: Dong, Zican, et al.
Published: (2024)

Unveiling the Flaws: Exploring Imperfections in Synthetic Data and Mitigation Strategies for Large Language Models
by: Chen, Jie, et al.
Published: (2024)

CAFE: Retrieval Head-based Coarse-to-Fine Information Seeking to Enhance Multi-Document QA Capability
by: Peng, Han, et al.
Published: (2025)

ShortGPT: Layers in Large Language Models are More Redundant Than You Expect
by: Men, Xin, et al.
Published: (2024)

ForesightKV: Optimizing KV Cache Eviction for Reasoning Models by Learning Long-Term Contribution
by: Dong, Zican, et al.
Published: (2026)

KV Shifting Attention Enhances Language Modeling
by: Xu, Mingyu, et al.
Published: (2024)

Towards Event-oriented Long Video Understanding
by: Du, Yifan, et al.
Published: (2024)

Base of RoPE Bounds Context Length
by: Men, Xin, et al.
Published: (2024)

Extracting and Combining Abilities For Building Multi-lingual Ability-enhanced Large Language Models
by: Chen, Zhipeng, et al.
Published: (2024)

LongPO: Long Context Self-Evolution of Large Language Models through Short-to-Long Preference Optimization
by: Chen, Guanzheng, et al.
Published: (2025)

Sticker-TTS: Learn to Utilize Historical Experience with a Sticker-driven Test-Time Scaling Framework
by: Chen, Jie, et al.
Published: (2025)

MetaGPT: Merging Large Language Models Using Model Exclusive Task Arithmetic
by: Zhou, Yuyan, et al.
Published: (2024)

SEGMENT+: Long Text Processing with Short-Context Language Models
by: Shi, Wei, et al.
Published: (2024)

ManuSearch: Democratizing Deep Search in Large Language Models with a Transparent and Open Multi-Agent Framework
by: Huang, Lisheng, et al.
Published: (2025)

How Efficient Are Diffusion Language Models? A Critical Examination of Efficiency Evaluation Practices
by: Peng, Han, et al.
Published: (2025)

UniPrefill: Universal Long-Context Prefill Acceleration via Block-wise Dynamic Sparsification
by: Fan, Qihang, et al.
Published: (2026)

Full-ECE: A Metric For Token-level Calibration on Large Language Models
by: Liu, Han, et al.
Published: (2024)

Short Data, Long Context: Distilling Positional Knowledge in Transformers
by: Huber, Patrick, et al.
Published: (2026)

Exploring the Design Space of Visual Context Representation in Video MLLMs
by: Du, Yifan, et al.
Published: (2024)

Mix-CPT: A Domain Adaptation Framework via Decoupling Knowledge Learning and Format Alignment
by: Jiang, Jinhao, et al.
Published: (2024)

LongAlign: A Recipe for Long Context Alignment of Large Language Models
by: Bai, Yushi, et al.
Published: (2024)

Domain-Specific Pruning of Large Mixture-of-Experts Models with Few-shot Demonstrations
by: Dong, Zican, et al.
Published: (2025)

YuLan-Mini: An Open Data-efficient Language Model
by: Hu, Yiwen, et al.
Published: (2024)

Think More, Hallucinate Less: Mitigating Hallucinations via Dual Process of Fast and Slow Thinking
by: Cheng, Xiaoxue, et al.
Published: (2025)

NeedleInATable: Exploring Long-Context Capability of Large Language Models towards Long-Structured Tables
by: Wang, Lanrui, et al.
Published: (2025)

Combining On-Policy Optimization and Distillation for Long-Context Reasoning in Large Language Models
by: Ramos, Miguel Moura, et al.
Published: (2026)

LongRecipe: Recipe for Efficient Long Context Generalization in Large Language Models
by: Hu, Zhiyuan, et al.
Published: (2024)

IUQ: Interrogative Uncertainty Quantification for Long-Form Large Language Model Generation
by: Fan, Haozhi, et al.
Published: (2026)

LongSafety: Evaluating Long-Context Safety of Large Language Models
by: Lu, Yida, et al.
Published: (2025)

FlashPrefill: Instantaneous Pattern Discovery and Thresholding for Ultra-Fast Long-Context Prefilling
by: Fan, Qihang, et al.
Published: (2026)

Mitigating Long-tail Distribution in Oracle Bone Inscriptions: Dataset, Model, and Benchmark
by: Li, Jinhao, et al.
Published: (2025)

LongLoRA: Efficient Fine-tuning of Long-Context Large Language Models
by: Chen, Yukang, et al.
Published: (2023)

Long-Short Alignment for Effective Long-Context Modeling in LLMs
by: Du, Tianqi, et al.
Published: (2025)

ChainLM: Empowering Large Language Models with Improved Chain-of-Thought Prompting
by: Cheng, Xiaoxue, et al.
Published: (2024)

OPSDL: On-Policy Self-Distillation for Long-Context Language Models
by: Zhang, Xinsen, et al.
Published: (2026)

ReSum: Unlocking Long-Horizon Search Intelligence via Context Summarization
by: Wu, Xixi, et al.
Published: (2025)

A Survey of Large Language Models
by: Zhao, Wayne Xin, et al.
Published: (2023)

Unlocking General Long Chain-of-Thought Reasoning Capabilities of Large Language Models via Representation Engineering
by: Tang, Xinyu, et al.
Published: (2025)