How to Synthesize Text Data without Model Collapse?

Fuente: arXiv

Saved in:

Bibliographic Details
Main Authors:	Zhu, Xuekai, Cheng, Daixuan, Li, Hengli, Zhang, Kaiyan, Hua, Ermo, Lv, Xingtai, Ding, Ning, Lin, Zhouhan, Zheng, Zilong, Zhou, Bowen
Format:	Preprint
Published:	2024
Subjects:	Computation and Language Artificial Intelligence Machine Learning
Online Access:	Acceder al recurso
Tags:	Add Tag No Tags, Be the first to tag this record!

Similar Items

Scalable Efficient Training of Large Language Models with Low-dimensional Projected Attention
by: Lv, Xingtai, et al.
Published: (2024)

Fast and Slow Generating: An Empirical Study on Large and Small Language Models Collaborative Decoding
by: Zhang, Kaiyan, et al.
Published: (2024)

Intuitive Fine-Tuning: Towards Simplifying Alignment into a Single Process
by: Hua, Ermo, et al.
Published: (2024)

Fourier Position Embedding: Enhancing Attention's Periodic Extension for Length Generalization
by: Hua, Ermo, et al.
Published: (2024)

Technologies on Effectiveness and Efficiency: A Survey of State Spaces Models
by: Lv, Xingtai, et al.
Published: (2025)

FlowRL: Matching Reward Distributions for LLM Reasoning
by: Zhu, Xuekai, et al.
Published: (2025)

PaD: Program-aided Distillation Can Teach Small Models Reasoning Better than Chain-of-thought Fine-tuning
by: Zhu, Xuekai, et al.
Published: (2023)

Critical Data Size of Language Models from a Grokking Perspective
by: Zhu, Xuekai, et al.
Published: (2024)

CoGenesis: A Framework Collaborating Large and Small Language Models for Secure Context-Aware Instruction Following
by: Zhang, Kaiyan, et al.
Published: (2024)

UltraMedical: Building Specialized Generalists in Biomedicine
by: Zhang, Kaiyan, et al.
Published: (2024)

MedXpertQA: Benchmarking Expert-Level Medical Reasoning and Understanding
by: Zuo, Yuxin, et al.
Published: (2025)

Towards a Unified View of Large Language Model Post-Training
by: Lv, Xingtai, et al.
Published: (2025)

Retrieval-Augmented Visual Question Answering via Built-in Autoregressive Search Engines
by: Long, Xinwei, et al.
Published: (2025)

Synthesizing Privacy-Preserving Text Data via Finetuning without Finetuning Billion-Scale LLMs
by: Tan, Bowen, et al.
Published: (2025)

Mastering Text, Code and Math Simultaneously via Fusing Highly Specialized Language Models
by: Ding, Ning, et al.
Published: (2024)

How Far Can Unsupervised RLVR Scale LLM Training?
by: He, Bingxiang, et al.
Published: (2026)

TTRL: Test-Time Reinforcement Learning
by: Zuo, Yuxin, et al.
Published: (2025)

Reasoning with Exploration: An Entropy Perspective
by: Cheng, Daixuan, et al.
Published: (2025)

Large Language Models as Biomedical Hypothesis Generators: A Comprehensive Evaluation
by: Qi, Biqing, et al.
Published: (2024)

MindDial: Belief Dynamics Tracking with Theory-of-Mind Modeling for Situated Neural Dialogue Generation
by: Qiu, Shuwen, et al.
Published: (2023)

Free Process Rewards without Process Labels
by: Yuan, Lifan, et al.
Published: (2024)

Towards Building Specialized Generalist AI with System 1 and System 2 Fusion
by: Zhang, Kaiyan, et al.
Published: (2024)

Discrete Markov Bridge
by: Li, Hengli, et al.
Published: (2025)

How Do Different Forms of Note‐Taking Affect Second Language Vocabulary Learning?
by: Zhouhan Jin, et al.
Published: (2025)

LFQA-E: Carefully Benchmarking Long-form QA Evaluation
by: Fan, Yuchen, et al.
Published: (2024)

Adapting Large Language Models to Domains via Reading Comprehension
by: Cheng, Daixuan, et al.
Published: (2023)

Seek in the Dark: Reasoning via Test-Time Instance-Level Policy Gradient in Latent Space
by: Li, Hengli, et al.
Published: (2025)

Synthesizing Text-to-SQL Data from Weak and Strong LLMs
by: Yang, Jiaxi, et al.
Published: (2024)

Towards Controlled Table-to-Text Generation with Scientific Reasoning
by: Guo, Zhixin, et al.
Published: (2023)

Post-Trained MoE Can Skip Half Experts via Self-Distillation
by: Lv, Xingtai, et al.
Published: (2026)

SSRL: Self-Search Reinforcement Learning
by: Fan, Yuchen, et al.
Published: (2025)

MLP Memory: A Retriever-Pretrained Memory for Large Language Models
by: Wei, Rubin, et al.
Published: (2025)

Enhancing Cross-lingual Sentence Embedding for Low-resource Languages with Word Alignment
by: Miao, Zhongtao, et al.
Published: (2024)

Memory Decoder: A Pretrained, Plug-and-Play Memory for Large Language Models
by: Cao, Jiaqi, et al.
Published: (2025)

Training LLMs to be Better Text Embedders through Bidirectional Reconstruction
by: Su, Chang, et al.
Published: (2025)

QChunker: Learning Question-Aware Text Chunking for Domain RAG via Multi-Agent Debate
by: Zhao, Jihao, et al.
Published: (2026)

Training-free LLM-generated Text Detection by Mining Token Probability Sequences
by: Xu, Yihuai, et al.
Published: (2024)

Automating Exploratory Proteomics Research via Language Models
by: Ding, Ning, et al.
Published: (2024)

Automating Exploratory Multiomics Research via Language Models
by: Qu, Shang, et al.
Published: (2025)

How Efficient Are Diffusion Language Models? A Critical Examination of Efficiency Evaluation Practices
by: Peng, Han, et al.
Published: (2025)