Copertina

Untie the Knots: An Efficient Data Augmentation Strategy for Long-Context Pre-Training in Language Models

Fuente: arXiv

Salvato in:

Dettagli Bibliografici
Autori principali:	Tian, Junfeng, Zheng, Da, Cheng, Yang, Wang, Rui, Zhang, Colin, Zhang, Debing
Natura:	Preprint
Pubblicazione:	2024
Soggetti:	Computation and Language Artificial Intelligence
Accesso online:	Acceder al recurso
Tags:	Aggiungi Tag Nessun Tag, puoi essere il primo ad aggiungerne!!

Documenti analoghi

NExtLong: Toward Effective Long-Context Training without Long Documents
di: Gao, Chaochen, et al.
Pubblicazione: (2025)

LiteLong: Resource-Efficient Long-Context Data Synthesis for LLMs
di: Jia, Junlong, et al.
Pubblicazione: (2025)

EntropyLong: Effective Long-Context Training via Predictive Uncertainty
di: Jia, Junlong, et al.
Pubblicazione: (2025)

LongSkywork: A Training Recipe for Efficiently Extending Context Length in Large Language Models
di: Zhao, Liang, et al.
Pubblicazione: (2024)

LooGLE: Can Long-Context Language Models Understand Long Contexts?
di: Li, Jiaqi, et al.
Pubblicazione: (2023)

LongBench Pro: A More Realistic and Comprehensive Bilingual Long-Context Evaluation Benchmark
di: Chen, Ziyang, et al.
Pubblicazione: (2026)

One Weird Trick to Untie Landin's Knot
di: Koronkevich, Paulette, et al.
Pubblicazione: (2025)

AdaComp: Extractive Context Compression with Adaptive Predictor for Retrieval-Augmented Large Language Models
di: Zhang, Qianchi, et al.
Pubblicazione: (2024)

LongMagpie: A Self-synthesis Method for Generating Large-scale Long-context Instructions
di: Gao, Chaochen, et al.
Pubblicazione: (2025)

CodePMP: Scalable Preference Model Pretraining for Large Language Model Reasoning
di: Yu, Huimu, et al.
Pubblicazione: (2024)

Context-Aware Counterfactual Data Augmentation for Gender Bias Mitigation in Language Models
di: Parihar, Shweta, et al.
Pubblicazione: (2026)

MOM: Memory-Efficient Offloaded Mini-Sequence Inference for Long Context Language Models
di: Zhang, Junyang, et al.
Pubblicazione: (2025)

Modular Techniques for Synthetic Long-Context Data Generation in Language Model Training and Evaluation
di: Subramanian, Seganrasan, et al.
Pubblicazione: (2025)

LongSafety: Evaluating Long-Context Safety of Large Language Models
di: Lu, Yida, et al.
Pubblicazione: (2025)

Scalable In-Context Learning on Tabular Data via Retrieval-Augmented Large Language Models
di: Wen, Xumeng, et al.
Pubblicazione: (2025)

MMRAG: Multi-Mode Retrieval-Augmented Generation with Large Language Models for Biomedical In-Context Learning
di: Zhan, Zaifu, et al.
Pubblicazione: (2025)

Document-Level In-Context Few-Shot Relation Extraction via Pre-Trained Language Models
di: Ozyurt, Yilmazcan, et al.
Pubblicazione: (2023)

RedStar: Does Scaling Long-CoT Data Unlock Better Slow-Reasoning Systems?
di: Xu, Haotian, et al.
Pubblicazione: (2025)

InfLLM: Training-Free Long-Context Extrapolation for LLMs with an Efficient Context Memory
di: Xiao, Chaojun, et al.
Pubblicazione: (2024)

OPSDL: On-Policy Self-Distillation for Long-Context Language Models
di: Zhang, Xinsen, et al.
Pubblicazione: (2026)

SoftDedup: an Efficient Data Reweighting Method for Speeding Up Language Model Pre-training
di: He, Nan, et al.
Pubblicazione: (2024)

How Training Data Shapes the Use of Parametric and In-Context Knowledge in Language Models
di: Kim, Minsung, et al.
Pubblicazione: (2025)

DEFT: Data Efficient Fine-Tuning for Pre-Trained Language Models via Unsupervised Core-Set Selection
di: Das, Devleena, et al.
Pubblicazione: (2023)

MemoRAG: Boosting Long Context Processing with Global Memory-Enhanced Retrieval Augmentation
di: Qian, Hongjin, et al.
Pubblicazione: (2024)

Embedding-to-Prefix: Parameter-Efficient Personalization for Pre-Trained Large Language Models
di: Huber, Bernd, et al.
Pubblicazione: (2025)

Legal Documents Drafting with Fine-Tuned Pre-Trained Large Language Model
di: Lin, Chun-Hsien, et al.
Pubblicazione: (2024)

"According to ...": Prompting Language Models Improves Quoting from Pre-Training Data
di: Weller, Orion, et al.
Pubblicazione: (2023)

Parameters vs. Context: Fine-Grained Control of Knowledge Reliance in Language Models
di: Bi, Baolong, et al.
Pubblicazione: (2025)

From 128K to 4M: Efficient Training of Ultra-Long Context Large Language Models
di: Xu, Chejian, et al.
Pubblicazione: (2025)

SemToken: Semantic-Aware Tokenization for Efficient Long-Context Language Modeling
di: Liu, Dong, et al.
Pubblicazione: (2025)

Context-DPO: Aligning Language Models for Context-Faithfulness
di: Bi, Baolong, et al.
Pubblicazione: (2024)

Evolution of Concepts in Language Model Pre-Training
di: Ge, Xuyang, et al.
Pubblicazione: (2025)

G-MemLLM: Gated Latent Memory Augmentation for Long-Context Reasoning in Large Language Models
di: Xu, Xun
Pubblicazione: (2026)

InftyThink: Breaking the Length Limits of Long-Context Reasoning in Large Language Models
di: Yan, Yuchen, et al.
Pubblicazione: (2025)

Efficient Long CoT Reasoning in Small Language Models
di: Wang, Zhaoyang, et al.
Pubblicazione: (2025)

Retrieval Augmented Generation or Long-Context LLMs? A Comprehensive Study and Hybrid Approach
di: Li, Zhuowan, et al.
Pubblicazione: (2024)

ACC: Compiling Agent Trajectories for Long-Context Training
di: Su, Qisheng, et al.
Pubblicazione: (2026)

LaRA: Benchmarking Retrieval-Augmented Generation and Long-Context LLMs -- No Silver Bullet for LC or RAG Routing
di: Li, Kuan, et al.
Pubblicazione: (2025)

MiniCPM-SALA: Hybridizing Sparse and Linear Attention for Efficient Long-Context Modeling
di: MiniCPM Team, et al.
Pubblicazione: (2026)

Efficient Length-Generalizable Attention via Causal Retrieval for Long-Context Language Modeling
di: Hu, Xiang, et al.
Pubblicazione: (2024)