Cover Image

Rethinking KenLM: Good and Bad Model Ensembles for Efficient Text Quality Filtering in Large Web Corpora

Fuente: arXiv

Saved in:

Bibliographic Details
Main Authors:	Kim, Yungi, Ha, Hyunsoo, Lee, Sukyung, Kim, Jihoo, Yang, Seonghoon, Park, Chanjun
Format:	Preprint
Published:	2024
Subjects:	Computation and Language Artificial Intelligence
Online Access:	Acceder al recurso
Tags:	Add Tag No Tags, Be the first to tag this record!

Similar Items

LP Data Pipeline: Lightweight, Purpose-driven Data Pipeline for Large Language Models
by: Kim, Yungi, et al.
Published: (2024)

1 Trillion Token (1TT) Platform: A Novel Framework for Efficient Data Sharing and Compensation in Large Language Models
by: Park, Chanjun, et al.
Published: (2024)

Open Ko-LLM Leaderboard2: Bridging Foundational and Practical Evaluation for Korean LLMs
by: Kim, Hyeonwoo, et al.
Published: (2024)

Representing the Under-Represented: Cultural and Core Capability Benchmarks for Developing Thai Large Language Models
by: Kim, Dahyun, et al.
Published: (2024)

Dataverse: Open-Source ETL (Extract, Transform, Load) Pipeline for Large Language Models
by: Park, Hyunbyung, et al.
Published: (2024)

Evalverse: Unified and Accessible Library for Large Language Model Evaluation
by: Kim, Jihoo, et al.
Published: (2024)

SAAS: Solving Ability Amplification Strategy for Enhanced Mathematical Reasoning in Large Language Models
by: Kim, Hyeonwoo, et al.
Published: (2024)

InstaTrans: An Instruction-Aware Translation Framework for Non-English Instruction Datasets
by: Kim, Yungi, et al.
Published: (2024)

Open Ko-LLM Leaderboard: Evaluating Large Language Models in Korean with Ko-H5 Benchmark
by: Park, Chanjun, et al.
Published: (2024)

SOLAR 10.7B: Scaling Large Language Models with Simple yet Effective Depth Up-Scaling
by: Kim, Dahyun, et al.
Published: (2023)

sDPO: Don't Use Your Data All at Once
by: Kim, Dahyun, et al.
Published: (2024)

Beyond Line-Level Filtering for the Pretraining Corpora of LLMs
by: Park, Chanwoo, et al.
Published: (2025)

Can Code-Switched Texts Activate a Knowledge Switch in LLMs? A Case Study on English-Korean Code-Switching
by: Kim, Seoyeon, et al.
Published: (2024)

Understanding LLM Development Through Longitudinal Study: Insights from the Open Ko-LLM Leaderboard
by: Park, Chanjun, et al.
Published: (2024)

KITE: A Benchmark for Evaluating Korean Instruction-Following Abilities in Large Language Models
by: Kim, Dongjun, et al.
Published: (2025)

KVoiceBench, KOpenAudioBench, and KMMAU: Agent-Driven Korean Speech Benchmarks for Evaluating SpeechLMs
by: Kim, Haechan, et al.
Published: (2026)

Enhancing Automatic Term Extraction with Large Language Models via Syntactic Retrieval
by: Chun, Yongchan, et al.
Published: (2025)

Model-Based Data-Centric AI: Bridging the Divide Between Academic Ideals and Industrial Pragmatism
by: Park, Chanjun, et al.
Published: (2024)

LegalSearchLM: Rethinking Legal Case Retrieval as Legal Elements Generation
by: Kim, Chaeeun, et al.
Published: (2025)

Findings of the BabyLM Challenge: Sample-Efficient Pretraining on Developmentally Plausible Corpora
by: Warstadt, Alex, et al.
Published: (2025)

When Should Dense Retrievers Be Updated in Evolving Corpora? Detecting Out-of-Distribution Corpora Using GradNormIR
by: Ko, Dayoon, et al.
Published: (2025)

Enhancing Document-Level Machine Translation via Filtered Synthetic Corpora and Two-Stage LLM Adaptation
by: Kim, Ireh, et al.
Published: (2026)

Findings of the Second BabyLM Challenge: Sample-Efficient Pretraining on Developmentally Plausible Corpora
by: Hu, Michael Y., et al.
Published: (2024)

Do Language Models Care About Text Quality? Evaluating Web-Crawled Corpora Across 11 Languages
by: van Noord, Rik, et al.
Published: (2024)

Investigating the Influence of Prompt-Specific Shortcuts in AI Generated Text Detection
by: Park, Choonghyun, et al.
Published: (2024)

LANGSAE EDITING: Improving Multilingual Information Retrieval via Post-hoc Language Identity Removal
by: Kim, Dongjun, et al.
Published: (2026)

Exploring Coding Spot: Understanding Parametric Contributions to LLM Coding Performance
by: Kim, Dongjun, et al.
Published: (2024)

"Global is Good, Local is Bad?": Understanding Brand Bias in LLMs
by: Kamruzzaman, Mahammed, et al.
Published: (2024)

Quality Does Matter: A Detailed Look at the Quality and Utility of Web-Mined Parallel Corpora
by: Ranathunga, Surangika, et al.
Published: (2024)

Fair Play in the Newsroom: Actor-Based Filtering Gender Discrimination in Text Corpora
by: Urchs, Stefanie, et al.
Published: (2025)

FLEX: A Benchmark for Evaluating Robustness of Fairness in Large Language Models
by: Jung, Dahyun, et al.
Published: (2025)

Benchmark Profiling: Mechanistic Diagnosis of LLM Benchmarks
by: Kim, Dongjun, et al.
Published: (2025)

EvalLM: Interactive Evaluation of Large Language Model Prompts on User-Defined Criteria
by: Kim, Tae Soo, et al.
Published: (2023)

LCIRC: A Recurrent Compression Approach for Efficient Long-form Context and Query Dependent Modeling in LLMs
by: An, Sumin, et al.
Published: (2025)

ChatLang-8: An LLM-Based Synthetic Data Generation Framework for Grammatical Error Correction
by: Park, Jeiyoon, et al.
Published: (2024)

CharacterGPT: A Persona Reconstruction Framework for Role-Playing Agents
by: Park, Jeiyoon, et al.
Published: (2024)

LegiGPT: Party Politics and Transport Policy with Large Language Model
by: Yun, Hyunsoo, et al.
Published: (2025)

KiC: Keyword-inspired Cascade for Cost-Efficient Text Generation with LLMs
by: Kim, Woo-Chan, et al.
Published: (2025)

OWQ: Outlier-Aware Weight Quantization for Efficient Fine-Tuning and Inference of Large Language Models
by: Lee, Changhun, et al.
Published: (2023)

Adaptive Contrastive Decoding in Retrieval-Augmented Generation for Handling Noisy Contexts
by: Kim, Youna, et al.
Published: (2024)