Cover Image

Clean First, Align Later: Benchmarking Preference Data Cleaning for Reliable LLM Alignment

Fuente: arXiv

Saved in:

Bibliographic Details
Main Authors:	Yeh, Samuel, Li, Sharon
Format:	Preprint
Published:	2025
Subjects:	Artificial Intelligence Computation and Language
Online Access:	Acceder al recurso
Tags:	Add Tag No Tags, Be the first to tag this record!

Similar Items

Token Cleaning: Fine-Grained Data Selection for LLM Supervised Fine-Tuning
by: Pang, Jinlong, et al.
Published: (2025)

EchoDistill:Alignment Noisy-to-Clean Self-Distillation for Robust Audio LLMs
by: Lin, Liang, et al.
Published: (2026)

Auditing Stealth Sycophancy in Mental-Health Dialogue: Structured Clinical-State Diagnostics and Clean Matched Benchmarks
by: Han, Tianze, et al.
Published: (2026)

CleanPatrick: A Benchmark for Image Data Cleaning
by: Gröger, Fabian, et al.
Published: (2025)

AlpsBench: An LLM Personalization Benchmark for Real-Dialogue Memorization and Preference Alignment
by: Xiao, Jianfei, et al.
Published: (2026)

The KL3M Data Project: Copyright-Clean Training Resources for Large Language Models
by: Bommarito II, Michael J, et al.
Published: (2025)

CleanGraph: Human-in-the-loop Knowledge Graph Refinement and Completion
by: Bikaun, Tyler, et al.
Published: (2024)

ConceptRM: The Quest to Mitigate Alert Fatigue through Consensus-Based Purity-Driven Data Cleaning for Reflection Modelling
by: Yu, Yongda, et al.
Published: (2026)

Enhancing Grammatical Error Detection using BERT with Cleaned Lang-8 Dataset
by: Nihalani, Rahul, et al.
Published: (2024)

LLM-Powered Benchmark Factory: Reliable, Generic, and Efficient
by: Yuan, Peiwen, et al.
Published: (2025)

MathClean: A Benchmark for Synthetic Mathematical Data Cleaning
by: Liang, Hao, et al.
Published: (2025)

Capturing Nuanced Preferences: Preference-Aligned Distillation for Small Language Models
by: Gu, Yanggan, et al.
Published: (2025)

Safer-Instruct: Aligning Language Models with Automated Preference Data
by: Shi, Taiwei, et al.
Published: (2023)

Less is More: Improving LLM Alignment via Preference Data Selection
by: Deng, Xun, et al.
Published: (2025)

TriAlign: Towards Universal Truth Consistency in Personalized LLM Alignment
by: Nguyen, Thi-Nhung, et al.
Published: (2026)

The Multilingual Alignment Prism: Aligning Global and Local Preferences to Reduce Harm
by: Aakanksha, et al.
Published: (2024)

Can LLMs Clean Up Your Mess? A Survey of Application-Ready Data Preparation with LLMs
by: Zhou, Wei, et al.
Published: (2026)

PMMT: Preference Alignment in Multilingual Machine Translation via LLM Distillation
by: Sun, Shuqiao, et al.
Published: (2024)

Pref-CTRL: Preference Driven LLM Alignment using Representation Editing
by: Ashrafi, Imranul, et al.
Published: (2026)

A Highly Clean Recipe Dataset with Ingredient States Annotation for State Probing Task
by: Toyooka, Mashiro, et al.
Published: (2025)

LLM-Align: Utilizing Large Language Models for Entity Alignment in Knowledge Graphs
by: Chen, Xuan, et al.
Published: (2024)

Preference Ranking Optimization for Human Alignment
by: Song, Feifan, et al.
Published: (2023)

TODO: Enhancing LLM Alignment with Ternary Preferences
by: Guo, Yuxiang, et al.
Published: (2024)

Spread Preference Annotation: Direct Preference Judgment for Efficient LLM Alignment
by: Kim, Dongyoung, et al.
Published: (2024)

LifeAlign: Lifelong Alignment for Large Language Models with Memory-Augmented Focalized Preference Optimization
by: Li, Junsong, et al.
Published: (2025)

Disentangling Preference Representation and Text Generation for Efficient Individual Preference Alignment
by: Zhang, Jianfei, et al.
Published: (2024)

From Meta-Thought to Execution: Cognitively Aligned Post-Training for Generalizable and Reliable LLM Reasoning
by: Wang, Shaojie, et al.
Published: (2026)

AlignBench: Benchmarking Chinese Alignment of Large Language Models
by: Liu, Xiao, et al.
Published: (2023)

Is On-Policy Data always the Best Choice for Direct Preference Optimization-based LM Alignment?
by: Sun, Zetian, et al.
Published: (2025)

Aligning Large Language Models with Searcher Preferences
by: Wu, Wei, et al.
Published: (2026)

Magpie: Alignment Data Synthesis from Scratch by Prompting Aligned LLMs with Nothing
by: Xu, Zhangchen, et al.
Published: (2024)

Acquiring Clean Language Models from Backdoor Poisoned Datasets by Downscaling Frequency Space
by: Wu, Zongru, et al.
Published: (2024)

Adversarial Preference Optimization: Enhancing Your Alignment via RM-LLM Game
by: Cheng, Pengyu, et al.
Published: (2023)

CURATRON: Complete and Robust Preference Data for Rigorous Alignment of Large Language Models
by: Nguyen, Son The, et al.
Published: (2024)

ALIGN: Prompt-based Attribute Alignment for Reliable, Responsible, and Personalized LLM-based Decision-Making
by: Ravichandran, Bharadwaj, et al.
Published: (2025)

Word Alignment as Preference for Machine Translation
by: Wu, Qiyu, et al.
Published: (2024)

Dissecting Human and LLM Preferences
by: Li, Junlong, et al.
Published: (2024)

Fake Alignment: Are LLMs Really Aligned Well?
by: Wang, Yixu, et al.
Published: (2023)

ASGEA: Exploiting Logic Rules from Align-Subgraphs for Entity Alignment
by: Luo, Yangyifei, et al.
Published: (2024)

More RLHF, More Trust? On The Impact of Preference Alignment On Trustworthiness
by: Li, Aaron J., et al.
Published: (2024)