How Bad is Training on Synthetic Data? A Statistical Analysis of Language Model Collapse

Fuente: arXiv

Salvato in:

Dettagli Bibliografici
Autori principali:	Seddik, Mohamed El Amine, Chen, Suei-Wen, Hayou, Soufiane, Youssef, Pierre, Debbah, Merouane
Natura:	Preprint
Pubblicazione:	2024
Soggetti:	Machine Learning Artificial Intelligence Computation and Language
Accesso online:	Acceder al recurso
Tags:	Aggiungi Tag Nessun Tag, puoi essere il primo ad aggiungerne!!

Documenti analoghi

Maximizing the Potential of Synthetic Data: Insights from Random Matrix Theory
di: Firdoussi, Aymane El, et al.
Pubblicazione: (2024)

A Proof of Learning Rate Transfer under $μ$P
di: Hayou, Soufiane
Pubblicazione: (2025)

On the Stability of the Jacobian Matrix in Deep Neural Networks
di: Dadoun, Benjamin, et al.
Pubblicazione: (2025)

Optimal Embedding Learning Rate in LLMs: The Effect of Vocabulary Size
di: Hayou, Soufiane, et al.
Pubblicazione: (2025)

LoRA+: Efficient Low Rank Adaptation of Large Models
di: Hayou, Soufiane, et al.
Pubblicazione: (2024)

Learning Rate Scaling across LoRA Ranks and Transfer to Full Finetuning
di: Chen, Nan, et al.
Pubblicazione: (2026)

How Small Can 6G Reason? Scaling Tiny Language Models for AI-Native Networks
di: Ferrag, Mohamed Amine, et al.
Pubblicazione: (2026)

The Impact of Initialization on LoRA Finetuning Dynamics
di: Hayou, Soufiane, et al.
Pubblicazione: (2024)

Reasoning Beyond Limits: Advances and Open Problems for LLMs
di: Ferrag, Mohamed Amine, et al.
Pubblicazione: (2025)

PLoP: Precise LoRA Placement for Efficient Finetuning of Large Models
di: Hayou, Soufiane, et al.
Pubblicazione: (2025)

6G-Bench: An Open Benchmark for Semantic Communication and Network-Level Reasoning with Foundation Models in AI-Native 6G Networks
di: Ferrag, Mohamed Amine, et al.
Pubblicazione: (2026)

UAVBench: An Open Benchmark Dataset for Autonomous and Agentic AI UAV Systems via LLM-Generated Flight Scenarios
di: Ferrag, Mohamed Amine, et al.
Pubblicazione: (2025)

AgentDrive: An Open Benchmark Dataset for Agentic AI Reasoning with LLM-Generated Scenarios in Autonomous Systems
di: Ferrag, Mohamed Amine, et al.
Pubblicazione: (2026)

TeleMath: A Benchmark for Large Language Models in Telecom Mathematical Problem Solving
di: Colle, Vincenzo, et al.
Pubblicazione: (2025)

$α^3$-Bench: A Unified Benchmark of Safety, Robustness, and Efficiency for LLM-Based UAV Agents over 6G Networks
di: Ferrag, Mohamed Amine, et al.
Pubblicazione: (2026)

$α^3$-SecBench: A Large-Scale Evaluation Suite of Security, Resilience, and Trust for LLM-based UAV Agents over 6G Networks
di: Ferrag, Mohamed Amine, et al.
Pubblicazione: (2026)

From LLM Reasoning to Autonomous AI Agents: A Comprehensive Review
di: Ferrag, Mohamed Amine, et al.
Pubblicazione: (2025)

How Does Attention Help? Insights from Random Matrices on Signal Recovery from Sequence Models
di: Seddik, Mohamed El Amine
Pubblicazione: (2026)

6G Needs Agents: Toward Agentic AI-Native Networks for Autonomous Intelligence
di: Ferrag, Mohamed Amine, et al.
Pubblicazione: (2026)

LIDSA: Cognitive Arbitration for Signal-Free Autonomous Intersection Management
di: Lakas, Abderrahmane, et al.
Pubblicazione: (2026)

High-dimensional Learning with Noisy Labels
di: Firdoussi, Aymane El, et al.
Pubblicazione: (2024)

The Myth of Expert Specialization in MoEs: Why Routing Reflects Geometry, Not Necessarily Domain Expertise
di: Wang, Xi, et al.
Pubblicazione: (2026)

Finite-Sample Analysis of the Monte Carlo Exploring Starts Algorithm for Reinforcement Learning
di: Chen, Suei-Wen, et al.
Pubblicazione: (2024)

Analysis of AdvFusion: Adapter-based Multilingual Learning for Code Large Language Models
di: Esmaeili, Amirreza, et al.
Pubblicazione: (2025)

Escaping Collapse: The Strength of Weak Data for Large Language Model Training
di: Amin, Kareem, et al.
Pubblicazione: (2025)

$α$-LoRA: Effective Fine-Tuning via Base Model Rescaling
di: Firdoussi, Aymane El, et al.
Pubblicazione: (2025)

Syntriever: How to Train Your Retriever with Synthetic Data from LLMs
di: Kim, Minsang, et al.
Pubblicazione: (2025)

Do Vision and Language Encoders Represent the World Similarly?
di: Maniparambil, Mayug, et al.
Pubblicazione: (2024)

Is It Good Data for Multilingual Instruction Tuning or Just Bad Multilingual Evaluation for Large Language Models?
di: Chen, Pinzhen, et al.
Pubblicazione: (2024)

How to Synthesize Text Data without Model Collapse?
di: Zhu, Xuekai, et al.
Pubblicazione: (2024)

When Bad Data Leads to Good Models
di: Li, Kenneth, et al.
Pubblicazione: (2025)

Faster and Lighter LLMs: A Survey on Current Challenges and Way Forward
di: Chavan, Arnav, et al.
Pubblicazione: (2024)

How secure is AI-generated Code: A Large-Scale Comparison of Large Language Models
di: Tihanyi, Norbert, et al.
Pubblicazione: (2024)

CyberMetric: A Benchmark Dataset based on Retrieval-Augmented Generation for Evaluating LLMs in Cybersecurity Knowledge
di: Tihanyi, Norbert, et al.
Pubblicazione: (2024)

Modular Techniques for Synthetic Long-Context Data Generation in Language Model Training and Evaluation
di: Subramanian, Seganrasan, et al.
Pubblicazione: (2025)

M-MiniGPT4: Multilingual VLLM Alignment via Translated Data
di: Han, Seung Hun, et al.
Pubblicazione: (2026)

Scaling Laws of Synthetic Data for Language Models
di: Qin, Zeyu, et al.
Pubblicazione: (2025)

Cross-lingual Collapse: How Language-Centric Foundation Models Shape Reasoning in Large Language Models
di: Park, Cheonbok, et al.
Pubblicazione: (2025)

BAQ: Efficient Bit Allocation Quantization for Large Language Models
di: Zhang, Chao, et al.
Pubblicazione: (2025)

Synthetic Query Generation using Large Language Models for Virtual Assistants
di: Sannigrahi, Sonal, et al.
Pubblicazione: (2024)