Cover Image

Scaling Instruction-Tuned LLMs to Million-Token Contexts via Hierarchical Synthetic Data Generation

Fuente: arXiv

Saved in:

Bibliographic Details
Main Authors:	He, Linda, Wang, Jue, Weber, Maurice, Zhu, Shang, Athiwaratkun, Ben, Zhang, Ce
Format:	Preprint
Published:	2025
Subjects:	Computation and Language Artificial Intelligence
Online Access:	Acceder al recurso
Tags:	Add Tag No Tags, Be the first to tag this record!

Similar Items

Systematic Evaluation of the Quality of Synthetic Clinical Notes Rephrased by LLMs at Million-Note Scale
by: Liu, Jinghui, et al.
Published: (2026)

Token Alignment via Character Matching for Subword Completion
by: Athiwaratkun, Ben, et al.
Published: (2024)

Generalizing From Short to Long: Effective Data Synthesis for Long-Context Instruction Tuning
by: Zhu, Wenhao, et al.
Published: (2025)

HIRAG: Hierarchical-Thought Instruction-Tuning Retrieval-Augmented Generation
by: Jiao, YiHan, et al.
Published: (2025)

GemmAr: Enhancing LLMs Through Arabic Instruction-Tuning
by: Chouikhi, Hasna, et al.
Published: (2024)

GraphGen: Enhancing Supervised Fine-Tuning for LLMs with Knowledge-Driven Synthetic Data Generation
by: Chen, Zihong, et al.
Published: (2025)

Data Diversification Methods In Alignment Enhance Math Performance In LLMs
by: Dokmeci, Berkan, et al.
Published: (2025)

Span-level Emotion-Cause-Category Triplet Extraction with Instruction Tuning LLMs and Data Augmentation
by: Li, Xiangju, et al.
Published: (2025)

Beyond a Million Tokens: Benchmarking and Enhancing Long-Term Memory in LLMs
by: Tavakoli, Mohammad, et al.
Published: (2025)

A Modular Approach for Clinical SLMs Driven by Synthetic Data with Pre-Instruction Tuning, Model Merging, and Clinical-Tasks Alignment
by: Corbeil, Jean-Philippe, et al.
Published: (2025)

Jailbreak Instruction-Tuned LLMs via end-of-sentence MLP Re-weighting
by: Luo, Yifan, et al.
Published: (2024)

Next-Token Prediction Task Assumes Optimal Data Ordering for LLM Training in Proof Generation
by: An, Chenyang, et al.
Published: (2024)

OpenMathInstruct-1: A 1.8 Million Math Instruction Tuning Dataset
by: Toshniwal, Shubham, et al.
Published: (2024)

One Token Away from Collapse: The Fragility of Instruction-Tuned Helpfulness
by: Potraghloo, Erfan Baghaei, et al.
Published: (2026)

Refract ICL: Rethinking Example Selection in the Era of Million-Token Models
by: Akula, Arjun R., et al.
Published: (2025)

Staircase Streaming for Low-Latency Multi-Agent Inference
by: Wang, Junlin, et al.
Published: (2025)

Toward Graph-Tokenizing Large Language Models with Reconstructive Graph Instruction Tuning
by: Zhang, Zhongjian, et al.
Published: (2026)

Unveiling the Impact of Coding Data Instruction Fine-Tuning on Large Language Models Reasoning
by: Zhang, Xinlu, et al.
Published: (2024)

Instruction Tuning with and without Context: Behavioral Shifts and Downstream Impact
by: Lee, Hyunji, et al.
Published: (2025)

ClimateChat: Designing Data and Methods for Instruction Tuning LLMs to Answer Climate Change Queries
by: Chen, Zhou, et al.
Published: (2025)

Know When To Fold 'Em: Token-Efficient LLM Synthetic Data Generation via Multi-Stage In-Flight Rejection
by: Chowdhury, Anjir Ahmed, et al.
Published: (2026)

Step-On-Feet Tuning: Scaling Self-Alignment of LLMs via Bootstrapping
by: Wang, Haoyu, et al.
Published: (2024)

How Reliable Are Automatic Evaluation Methods for Instruction-Tuned LLMs?
by: Doostmohammadi, Ehsan, et al.
Published: (2024)

When Does Divide and Conquer Work for Long Context LLM? A Noise Decomposition Framework
by: Xu, Zhen, et al.
Published: (2025)

Improving Natural Language Understanding for LLMs via Large-Scale Instruction Synthesis
by: Yuan, Lin, et al.
Published: (2025)

DiffLM: Controllable Synthetic Data Generation via Diffusion Language Models
by: Zhou, Ying, et al.
Published: (2024)

Token Cleaning: Fine-Grained Data Selection for LLM Supervised Fine-Tuning
by: Pang, Jinlong, et al.
Published: (2025)

Dynamic Context Evolution for Scalable Synthetic Data Generation
by: Lingo, Ryan, et al.
Published: (2026)

MathScale: Scaling Instruction Tuning for Mathematical Reasoning
by: Tang, Zhengyang, et al.
Published: (2024)

T-SHIRT: Token-Selective Hierarchical Data Selection for Instruction Tuning
by: Fu, Yanjun, et al.
Published: (2025)

Building Accurate Translation-Tailored LLMs with Language Aware Instruction Tuning
by: Zan, Changtong, et al.
Published: (2024)

Selective Reflection-Tuning: Student-Selected Data Recycling for LLM Instruction-Tuning
by: Li, Ming, et al.
Published: (2024)

TACOS: Open Tagging and Comparative Scoring for Instruction Fine-Tuning Data Selection
by: He, Xixiang, et al.
Published: (2025)

CorpusQA: A 10 Million Token Benchmark for Corpus-Level Analysis and Reasoning
by: Lu, Zhiyuan, et al.
Published: (2026)

Controllable Text Generation in the Instruction-Tuning Era
by: Ashok, Dhananjay, et al.
Published: (2024)

Configurable Preference Tuning with Rubric-Guided Synthetic Data
by: Gallego, Víctor
Published: (2025)

SelectIT: Selective Instruction Tuning for LLMs via Uncertainty-Aware Self-Reflection
by: Liu, Liangxin, et al.
Published: (2024)

EMCEE: Improving Multilingual Capability of LLMs via Bridging Knowledge and Reasoning with Extracted Synthetic Multilingual Context
by: Koo, Hamin, et al.
Published: (2025)

Federated Data-Efficient Instruction Tuning for Large Language Models
by: Qin, Zhen, et al.
Published: (2024)

Not All Documents Are What You Need for Extracting Instruction Tuning Data
by: Zhang, Chi, et al.
Published: (2025)