Image de couverture de livre

Scaling Instruction-Tuned LLMs to Million-Token Contexts via Hierarchical Synthetic Data Generation

Fuente: arXiv

Enregistré dans:

Détails bibliographiques
Auteurs principaux:	He, Linda, Wang, Jue, Weber, Maurice, Zhu, Shang, Athiwaratkun, Ben, Zhang, Ce
Format:	Preprint
Publié:	2025
Sujets:	Computation and Language Artificial Intelligence
Accès en ligne:	Acceder al recurso
Tags:	Ajouter un tag Pas de tags, Soyez le premier à ajouter un tag!

Documents similaires

Systematic Evaluation of the Quality of Synthetic Clinical Notes Rephrased by LLMs at Million-Note Scale
par: Liu, Jinghui, et autres
Publié: (2026)

Token Alignment via Character Matching for Subword Completion
par: Athiwaratkun, Ben, et autres
Publié: (2024)

Generalizing From Short to Long: Effective Data Synthesis for Long-Context Instruction Tuning
par: Zhu, Wenhao, et autres
Publié: (2025)

HIRAG: Hierarchical-Thought Instruction-Tuning Retrieval-Augmented Generation
par: Jiao, YiHan, et autres
Publié: (2025)

GemmAr: Enhancing LLMs Through Arabic Instruction-Tuning
par: Chouikhi, Hasna, et autres
Publié: (2024)

GraphGen: Enhancing Supervised Fine-Tuning for LLMs with Knowledge-Driven Synthetic Data Generation
par: Chen, Zihong, et autres
Publié: (2025)

Data Diversification Methods In Alignment Enhance Math Performance In LLMs
par: Dokmeci, Berkan, et autres
Publié: (2025)

Span-level Emotion-Cause-Category Triplet Extraction with Instruction Tuning LLMs and Data Augmentation
par: Li, Xiangju, et autres
Publié: (2025)

Beyond a Million Tokens: Benchmarking and Enhancing Long-Term Memory in LLMs
par: Tavakoli, Mohammad, et autres
Publié: (2025)

A Modular Approach for Clinical SLMs Driven by Synthetic Data with Pre-Instruction Tuning, Model Merging, and Clinical-Tasks Alignment
par: Corbeil, Jean-Philippe, et autres
Publié: (2025)

Jailbreak Instruction-Tuned LLMs via end-of-sentence MLP Re-weighting
par: Luo, Yifan, et autres
Publié: (2024)

Next-Token Prediction Task Assumes Optimal Data Ordering for LLM Training in Proof Generation
par: An, Chenyang, et autres
Publié: (2024)

OpenMathInstruct-1: A 1.8 Million Math Instruction Tuning Dataset
par: Toshniwal, Shubham, et autres
Publié: (2024)

One Token Away from Collapse: The Fragility of Instruction-Tuned Helpfulness
par: Potraghloo, Erfan Baghaei, et autres
Publié: (2026)

Refract ICL: Rethinking Example Selection in the Era of Million-Token Models
par: Akula, Arjun R., et autres
Publié: (2025)

Toward Graph-Tokenizing Large Language Models with Reconstructive Graph Instruction Tuning
par: Zhang, Zhongjian, et autres
Publié: (2026)

Unveiling the Impact of Coding Data Instruction Fine-Tuning on Large Language Models Reasoning
par: Zhang, Xinlu, et autres
Publié: (2024)

Instruction Tuning with and without Context: Behavioral Shifts and Downstream Impact
par: Lee, Hyunji, et autres
Publié: (2025)

ClimateChat: Designing Data and Methods for Instruction Tuning LLMs to Answer Climate Change Queries
par: Chen, Zhou, et autres
Publié: (2025)

Staircase Streaming for Low-Latency Multi-Agent Inference
par: Wang, Junlin, et autres
Publié: (2025)

Know When To Fold 'Em: Token-Efficient LLM Synthetic Data Generation via Multi-Stage In-Flight Rejection
par: Chowdhury, Anjir Ahmed, et autres
Publié: (2026)

Step-On-Feet Tuning: Scaling Self-Alignment of LLMs via Bootstrapping
par: Wang, Haoyu, et autres
Publié: (2024)

How Reliable Are Automatic Evaluation Methods for Instruction-Tuned LLMs?
par: Doostmohammadi, Ehsan, et autres
Publié: (2024)

When Does Divide and Conquer Work for Long Context LLM? A Noise Decomposition Framework
par: Xu, Zhen, et autres
Publié: (2025)

Improving Natural Language Understanding for LLMs via Large-Scale Instruction Synthesis
par: Yuan, Lin, et autres
Publié: (2025)

DiffLM: Controllable Synthetic Data Generation via Diffusion Language Models
par: Zhou, Ying, et autres
Publié: (2024)

Token Cleaning: Fine-Grained Data Selection for LLM Supervised Fine-Tuning
par: Pang, Jinlong, et autres
Publié: (2025)

Dynamic Context Evolution for Scalable Synthetic Data Generation
par: Lingo, Ryan, et autres
Publié: (2026)

MathScale: Scaling Instruction Tuning for Mathematical Reasoning
par: Tang, Zhengyang, et autres
Publié: (2024)

T-SHIRT: Token-Selective Hierarchical Data Selection for Instruction Tuning
par: Fu, Yanjun, et autres
Publié: (2025)

Building Accurate Translation-Tailored LLMs with Language Aware Instruction Tuning
par: Zan, Changtong, et autres
Publié: (2024)

Selective Reflection-Tuning: Student-Selected Data Recycling for LLM Instruction-Tuning
par: Li, Ming, et autres
Publié: (2024)

TACOS: Open Tagging and Comparative Scoring for Instruction Fine-Tuning Data Selection
par: He, Xixiang, et autres
Publié: (2025)

CorpusQA: A 10 Million Token Benchmark for Corpus-Level Analysis and Reasoning
par: Lu, Zhiyuan, et autres
Publié: (2026)

Controllable Text Generation in the Instruction-Tuning Era
par: Ashok, Dhananjay, et autres
Publié: (2024)

Configurable Preference Tuning with Rubric-Guided Synthetic Data
par: Gallego, Víctor
Publié: (2025)

SelectIT: Selective Instruction Tuning for LLMs via Uncertainty-Aware Self-Reflection
par: Liu, Liangxin, et autres
Publié: (2024)

EMCEE: Improving Multilingual Capability of LLMs via Bridging Knowledge and Reasoning with Extracted Synthetic Multilingual Context
par: Koo, Hamin, et autres
Publié: (2025)

Federated Data-Efficient Instruction Tuning for Large Language Models
par: Qin, Zhen, et autres
Publié: (2024)

Not All Documents Are What You Need for Extracting Instruction Tuning Data
par: Zhang, Chi, et autres
Publié: (2025)