Cover Image

A Review of the Challenges with Massive Web-mined Corpora Used in Large Language Models Pre-Training

Fuente: arXiv

Saved in:

Bibliographic Details
Main Authors:	Perełkiewicz, Michał, Poświata, Rafał
Format:	Preprint
Published:	2024
Subjects:	Computation and Language
Online Access:	Acceder al recurso
Tags:	Add Tag No Tags, Be the first to tag this record!

Similar Items

PL-MTEB: Polish Massive Text Embedding Benchmark
by: Poświata, Rafał, et al.
Published: (2024)

SMCLM: Semantically Meaningful Causal Language Modeling for Autoregressive Paraphrase Generation
by: Perełkiewicz, Michał, et al.
Published: (2025)

PIRB: A Comprehensive Benchmark of Polish Dense and Hybrid Text Retrieval Methods
by: Dadas, Sławomir, et al.
Published: (2024)

Evaluating Polish linguistic and cultural competency in large language models
by: Dadas, Sławomir, et al.
Published: (2025)

Unveiling Dual Quality in Product Reviews: An NLP-Based Approach
by: Poświata, Rafał, et al.
Published: (2025)

Long-Context Encoder Models for Polish Language Understanding
by: Dadas, Sławomir, et al.
Published: (2026)

Improving the quality of Web-mined Parallel Corpora of Low-Resource Languages using Debiasing Heuristics
by: Fernando, Aloka, et al.
Published: (2025)

A Recipe of Parallel Corpora Exploitation for Multilingual Large Language Models
by: Lin, Peiqin, et al.
Published: (2024)

Do Language Models Care About Text Quality? Evaluating Web-Crawled Corpora Across 11 Languages
by: van Noord, Rik, et al.
Published: (2024)

EgyBERT: A Large Language Model Pretrained on Egyptian Dialect Corpora
by: Qarah, Faisal
Published: (2024)

Exploring Forgetting in Large Language Model Pre-Training
by: Liao, Chonghua, et al.
Published: (2024)

A Survey on Multilingual Large Language Models: Corpora, Alignment, and Bias
by: Xu, Yuemei, et al.
Published: (2024)

Leveraging Large Language Models to Measure Gender Representation Bias in Gendered Language Corpora
by: Derner, Erik, et al.
Published: (2024)

Pre-Trained Language Models for Keyphrase Prediction: A Review
by: Umair, Muhammad, et al.
Published: (2024)

New Textual Corpora for Serbian Language Modeling
by: Škorić, Mihailo, et al.
Published: (2024)

SaudiBERT: A Large Language Model Pretrained on Saudi Dialect Corpora
by: Qarah, Faisal
Published: (2024)

CLASSLA-web: Comparable Web Corpora of South Slavic Languages Enriched with Linguistic and Genre Annotation
by: Ljubešić, Nikola, et al.
Published: (2024)

Advancing Event Forecasting through Massive Training of Large Language Models: Challenges, Solutions, and Broader Impacts
by: Lee, Sang-Woo, et al.
Published: (2025)

The Growing Gains and Pains of Iterative Web Corpora Crawling: Insights from South Slavic CLASSLA-web 2.0 Corpora
by: Pungeršek, Taja Kuzman, et al.
Published: (2026)

Synthetic Pre-Pre-Training Improves Language Model Robustness to Noisy Pre-Training Data
by: Guo, Xu, et al.
Published: (2026)

Rethinking KenLM: Good and Bad Model Ensembles for Efficient Text Quality Filtering in Large Web Corpora
by: Kim, Yungi, et al.
Published: (2024)

Validating and Exploring Large Geographic Corpora
by: Dunn, Jonathan
Published: (2024)

Exploring Large Language Models in Healthcare: Insights into Corpora Sources, Customization Strategies, and Evaluation Metrics
by: Yang, Shuqi, et al.
Published: (2025)

Identifying Emerging Concepts in Large Corpora
by: Ma, Sibo, et al.
Published: (2025)

Wiki Dumps to Training Corpora: South Slavic Case
by: Škorić, Mihailo, et al.
Published: (2026)

A Study on Hidden Layer Distillation for Large Language Model Pre-Training
by: Guigon, Maxime, et al.
Published: (2026)

How Do Large Language Models Learn Concepts During Continual Pre-Training?
by: Yao, Barry Menglong, et al.
Published: (2026)

The Pre-Training Study of Expanded-SPLADE Models on Web Document Titles
by: Kim, Hiun, et al.
Published: (2026)

Quality Does Matter: A Detailed Look at the Quality and Utility of Web-Mined Parallel Corpora
by: Ranathunga, Surangika, et al.
Published: (2024)

Preference Consistency Matters: Enhancing Preference Learning in Language Models with Automated Self-Curation of Training Corpora
by: Lee, JoonHo, et al.
Published: (2024)

Massive Activations in Large Language Models
by: Sun, Mingjie, et al.
Published: (2024)

From Outliers to Topics in Language Models: Anticipating Trends in News Corpora
by: Zve, Evangelia, et al.
Published: (2025)

Parallel Corpora for Machine Translation in Low-resource Indic Languages: A Comprehensive Review
by: Raja, Rahul, et al.
Published: (2025)

On the Interplay of Pre-Training, Mid-Training, and RL on Reasoning Language Models
by: Zhang, Charlie, et al.
Published: (2025)

Learning Dynamics in Continual Pre-Training for Large Language Models
by: Wang, Xingjin, et al.
Published: (2025)

Enhancing Translation Accuracy of Large Language Models through Continual Pre-Training on Parallel Data
by: Kondo, Minato, et al.
Published: (2024)

Negation: A Pink Elephant in the Large Language Models' Room?
by: Vrabcová, Tereza, et al.
Published: (2025)

Organize the Web: Constructing Domains Enhances Pre-Training Data Curation
by: Wettig, Alexander, et al.
Published: (2025)

MaLA-500: Massive Language Adaptation of Large Language Models
by: Lin, Peiqin, et al.
Published: (2024)

Legal Documents Drafting with Fine-Tuned Pre-Trained Large Language Model
by: Lin, Chun-Hsien, et al.
Published: (2024)