Beyond Model Collapse: Scaling Up with Synthesized Data Requires Verification

Fuente: arXiv

Saved in:

Bibliographic Details
Main Authors:	Feng, Yunzhen, Dohmatob, Elvis, Yang, Pu, Charton, Francois, Kempe, Julia
Format:	Preprint
Published:	2024
Subjects:	Machine Learning Artificial Intelligence
Online Access:	Acceder al recurso
Tags:	Add Tag No Tags, Be the first to tag this record!

Similar Items

A Tale of Tails: Model Collapse as a Change of Scaling Laws
by: Dohmatob, Elvis, et al.
Published: (2024)

Model Collapse Demystified: The Case of Regression
by: Dohmatob, Elvis, et al.
Published: (2024)

Strong Model Collapse
by: Dohmatob, Elvis, et al.
Published: (2024)

Emergent properties with repeated examples
by: Charton, François, et al.
Published: (2024)

Efficient Refusal Ablation in LLM through Optimal Transport
by: Nanfack, Geraldin, et al.
Published: (2026)

Iteration Head: A Mechanistic Study of Chain-of-Thought
by: Cabannes, Vivien, et al.
Published: (2024)

Learning the greatest common divisor: explaining transformer predictions
by: Charton, François
Published: (2023)

Int2Int: a framework for mathematics with transformers
by: Charton, François
Published: (2025)

Attacking Bayes: On the Adversarial Robustness of Bayesian Neural Networks
by: Feng, Yunzhen, et al.
Published: (2024)

Scaling Laws for Associative Memories
by: Cabannes, Vivien, et al.
Published: (2023)

Improving the Scaling Laws of Synthetic Data with Deliberate Practice
by: Askari-Hemmat, Reyhane, et al.
Published: (2025)

The Pitfalls of Memorization: When Memorization Hurts Generalization
by: Bayat, Reza, et al.
Published: (2024)

How to Synthesize Text Data without Model Collapse?
by: Zhu, Xuekai, et al.
Published: (2024)

Instruction Diversity Drives Generalization To Unseen Tasks
by: Zhang, Dylan, et al.
Published: (2024)

$\textbf{Only-IF}$:Revealing the Decisive Effect of Instruction Diversity on Generalization
by: Zhang, Dylan, et al.
Published: (2024)

Mission Impossible: A Statistical Perspective on Jailbreaking LLMs
by: Su, Jingtong, et al.
Published: (2024)

From Concepts to Components: Concept-Agnostic Attention Module Discovery in Transformers
by: Su, Jingtong, et al.
Published: (2025)

Scaling Up Data Parallelism in Decentralized Deep Learning
by: Xie, Bing, et al.
Published: (2025)

Delay, Plateau, or Collapse: Evaluating the Impact of Systematic Verification Error on RLVR
by: Egashira, Kazuki, et al.
Published: (2026)

auto-fpt: Automating Free Probability Theory Calculations for Machine Learning Theory
by: Subramonian, Arjun, et al.
Published: (2025)

PILAF: Optimal Human Preference Sampling for Reward Modeling
by: Feng, Yunzhen, et al.
Published: (2025)

Egalitarian Gradient Descent: A Simple Approach to Accelerated Grokking
by: Pasand, Ali Saheb, et al.
Published: (2025)

Scaling Context Requires Rethinking Attention
by: Gelada, Carles, et al.
Published: (2025)

Why Less is More (Sometimes): A Theory of Data Curation
by: Dohmatob, Elvis, et al.
Published: (2025)

Loong: Synthesize Long Chain-of-Thoughts at Scale through Verifiers
by: Huang, Xingyue, et al.
Published: (2025)

Beyond Scaling Law: A Data-Efficient Distillation Framework for Reasoning
by: Wu, Xiaojun, et al.
Published: (2025)

From Symbolic Tasks to Code Generation: Diversification Yields Better Task Performers
by: Zhang, Dylan, et al.
Published: (2024)

Reasoning on the Manifold: Bidirectional Consistency for Self-Verification in Diffusion Language Models
by: Ruan, Jiaoyang, et al.
Published: (2026)

Scaling Up Bayesian DAG Sampling
by: Nikzad, Daniele, et al.
Published: (2025)

Beyond Inference-Time Search: Reinforcement Learning Synthesizes Reusable Solvers
by: Massoudi, Soheyl, et al.
Published: (2026)

Generating Pragmatic Examples to Train Neural Program Synthesizers
by: Vaduguru, Saujas, et al.
Published: (2023)

Don't Waste Mistakes: Leveraging Negative RL-Groups via Confidence Reweighting
by: Feng, Yunzhen, et al.
Published: (2025)

What Characterizes Effective Reasoning? Revisiting Length, Review, and Structure of CoT
by: Feng, Yunzhen, et al.
Published: (2025)

TQL: Scaling Q-Functions with Transformers by Preventing Attention Collapse
by: Dong, Perry, et al.
Published: (2026)

Continued AI Scaling Requires Repeated Efficiency Doublings
by: Lu, Chien-Ping
Published: (2026)

Embedding Trust: Semantic Isotropy Predicts Nonfactuality in Long-Form Text Generation
by: Bhardwaj, Dhrupad, et al.
Published: (2025)

MIDUS: Memory-Infused Depth Up-Scaling
by: Kim, Taero, et al.
Published: (2025)

Sample, Scrutinize and Scale: Effective Inference-Time Search by Scaling Verification
by: Zhao, Eric, et al.
Published: (2025)

Multi-modal Synthetic Data Training and Model Collapse: Insights from VLMs and Diffusion Models
by: Hu, Zizhao, et al.
Published: (2025)

Soft Tokens, Hard Truths
by: Butt, Natasha, et al.
Published: (2025)