How Can We Synthesize High-Quality Pretraining Data? A Systematic Study of Prompt Design, Generator Model, and Source Data

Documenti analoghi

• The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scale
  di: Penedo, Guilherme, et al.
  Pubblicazione: (2024)
• FineWeb2: One Pipeline to Scale Them All -- Adapting Pre-Training Data Processing to Every Language
  di: Penedo, Guilherme, et al.
  Pubblicazione: (2025)
• SmolLM2: When Smol Goes Big -- Data-Centric Training of a Small Language Model
  di: Allal, Loubna Ben, et al.
  Pubblicazione: (2025)
• Scaling Laws and Compute-Optimal Training Beyond Fixed Training Durations
  di: Hägele, Alexander, et al.
  Pubblicazione: (2024)
• FineVision: Open Data Is All You Need
  di: Wiedmann, Luis, et al.
  Pubblicazione: (2025)