Image de couverture de livre

SEDD: Scalable and Efficient Dataset Deduplication with GPUs

Fuente: arXiv

Enregistré dans:

Détails bibliographiques
Auteurs principaux:	Son, Youngjun, Kim, Chaewon, Lee, Jaejin
Format:	Preprint
Publié:	2025
Sujets:	Computation and Language
Accès en ligne:	Acceder al recurso
Tags:	Ajouter un tag Pas de tags, Soyez le premier à ajouter un tag!

Documents similaires

KLAAD: Refining Attention Mechanisms to Reduce Societal Bias in Generative Language Models
par: Kim, Seorin, et autres
Publié: (2025)

Thunder-DeID: Accurate and Efficient De-identification Framework for Korean Court Judgments
par: Hahm, Sungeun, et autres
Publié: (2025)

FENCE: A Financial and Multimodal Jailbreak Detection Dataset
par: Kim, Mirae, et autres
Publié: (2026)

Thunder-LLM: Efficiently Adapting LLMs to Korean with Minimal Resources
par: Kim, Jinpyo, et autres
Publié: (2025)

Generative Deduplication For Socia Media Data Selection
par: Li, Xianming, et autres
Publié: (2024)

Assessing Socio-Cultural Alignment and Technical Safety of Sovereign LLMs
par: Chae, Kyubyung, et autres
Publié: (2025)

BankMathBench: A Benchmark for Numerical Reasoning in Banking Scenarios
par: Lee, Yunseung, et autres
Publié: (2026)

Deduplicating and Ranking Solution Programs for Suggesting Reference Solutions
par: Shirafuji, Atsushi, et autres
Publié: (2023)

Reasoning by Commented Code for Table Question Answering
par: Pyo, Seho, et autres
Publié: (2026)

Unsupervised Extractive Dialogue Summarization in Hyperdimensional Space
par: Park, Seongmin, et autres
Publié: (2024)

Models Know Models Best: Evaluation via Model-Preferred Formats
par: Lee, Joonhak, et autres
Publié: (2026)

FairDeDup: Detecting and Mitigating Vision-Language Fairness Disparities in Semantic Dataset Deduplication
par: Slyman, Eric, et autres
Publié: (2024)

LaDiMo: Layer-wise Distillation Inspired MoEfier
par: Kim, Sungyoon, et autres
Publié: (2024)

Thunder-KoNUBench: A Corpus-Aligned Benchmark for Korean Negation Understanding
par: Jung, Sungmok, et autres
Publié: (2026)

ExpGuard: LLM Content Moderation in Specialized Domains
par: Choi, Minseok, et autres
Publié: (2026)

Choices Speak Louder than Questions
par: Cho, Gyeongje, et autres
Publié: (2025)

Stress-Testing Emotional Support Models: Moving from Homogeneous to Diverse Help Seekers
par: Heo, Chaewon, et autres
Publié: (2026)

Bounded Hyperbolic Tangent: A Stable and Efficient Alternative to Pre-Layer Normalization in Large Language Models
par: Byun, Hoyoon, et autres
Publié: (2025)

Ko-PIQA: A Korean Physical Commonsense Reasoning Dataset with Cultural Context
par: Choi, Dasol, et autres
Publié: (2025)

Thunder-NUBench: A Benchmark for LLMs' Sentence-Level Negation Understanding
par: So, Yeonkyoung, et autres
Publié: (2025)

DuET: Dual Execution for Test Output Prediction with Generated Code and Pseudocode
par: Han, Hojae, et autres
Publié: (2026)

Merlin: Deterministic Byte-Exact Deduplication for Lossless Context Optimization in Large Language Model Inference
par: Schelpe, Sietse
Publié: (2026)

SparseAccelerate: Efficient Long-Context Inference for Mid-Range GPUs
par: Vo, James
Publié: (2024)

PILOT-Bench: A Benchmark for Legal Reasoning in the Patent Domain with IRAC-Aligned Classification Tasks
par: Jang, Yehoon, et autres
Publié: (2026)

Beyond Line-Level Filtering for the Pretraining Corpora of LLMs
par: Park, Chanwoo, et autres
Publié: (2025)

References Indeed Matter? Reference-Free Preference Optimization for Conversational Query Reformulation
par: Kim, Doyoung, et autres
Publié: (2025)

Byte-Exact Deduplication in Retrieval-Augmented Generation: A Three-Regime Empirical Analysis Across Public Benchmarks
par: Schelpe, Sietse
Publié: (2026)

PERC: Plan-As-Query Example Retrieval for Underrepresented Code Generation
par: Yoo, Jaeseok, et autres
Publié: (2024)

ArchCode: Incorporating Software Requirements in Code Generation with Large Language Models
par: Han, Hojae, et autres
Publié: (2024)

HetCCL: Accelerating LLM Training with Heterogeneous GPUs
par: Kim, Heehoon, et autres
Publié: (2026)

Not All Adapters Matter: Selective Adapter Freezing for Memory-Efficient Fine-Tuning of Language Models
par: Son, Hyegang, et autres
Publié: (2024)

Thunder-Tok: Minimizing Tokens per Word in Tokenizing Korean Texts for Generative Language Models
par: Cho, Gyeongje, et autres
Publié: (2025)

Pretraining A Large Language Model using Distributed GPUs: A Memory-Efficient Decentralized Paradigm
par: Zhang, Jinrui, et autres
Publié: (2026)

Evaluating Deduplication Techniques for Economic Research Paper Titles with a Focus on Semantic Similarity using NLP and LLMs
par: You, Doohee, et autres
Publié: (2024)

KoCoNovel: Annotated Dataset of Character Coreference in Korean Novels
par: Kim, Kyuhee, et autres
Publié: (2024)

Locate&Edit: Energy-based Text Editing for Efficient, Flexible, and Faithful Controlled Text Generation
par: Son, Hye Ryung, et autres
Publié: (2024)

Scalable Video-to-Dataset Generation for Cross-Platform Mobile Agents
par: Jang, Yunseok, et autres
Publié: (2025)

Pushing the Boundaries of Multiple Choice Evaluation to One Hundred Options
par: Lee, Nahyun, et autres
Publié: (2026)

SDS KoPub VDR: A Benchmark Dataset for Visual Document Retrieval in Korean Public Documents
par: Lee, Jaehoon, et autres
Publié: (2025)

Privacy-Preserving Data Deduplication for Enhancing Federated Learning of Language Models (Extended Version)
par: Abadi, Aydin, et autres
Publié: (2024)