Cover Image

Direct Density Ratio Optimization: A Statistically Consistent Approach to Aligning Large Language Models

Fuente: arXiv

Saved in:

Bibliographic Details
Main Authors:	Higuchi, Rei, Suzuki, Taiji
Format:	Preprint
Published:	2025
Subjects:	Machine Learning Computation and Language
Online Access:	Acceder al recurso
Tags:	Add Tag No Tags, Be the first to tag this record!

Similar Items

A Relative-Budget Theory for Reinforcement Learning with Verifiable Rewards in Large Language Model Reasoning
by: Wachi, Akifumi, et al.
Published: (2026)

Relative Density Ratio Optimization for Stable and Statistically Consistent Model Alignment
by: Takahashi, Hiroshi, et al.
Published: (2026)

When Does Metadata Conditioning (NOT) Work for Language Model Pre-Training? A Study with Context-Free Grammars
by: Higuchi, Rei, et al.
Published: (2025)

Degrees of Freedom for Linear Attention: Distilling Softmax Attention with Optimal Feature Efficiency
by: Nishikawa, Naoki, et al.
Published: (2025)

How Neural Reward Models Learn Features for Policy Optimization: A Single-Index Analysis
by: Higuchi, Rei, et al.
Published: (2026)

Vulnerability Mitigation for Safety-Aligned Language Models via Debiasing
by: Tran, Thien Q., et al.
Published: (2025)

Inference-Aware Meta-Alignment of LLMs via Non-Linear GRPO
by: Takakura, Shokichi, et al.
Published: (2026)

Temporally Consistent Factuality Probing for Large Language Models
by: Bajpai, Ashutosh, et al.
Published: (2024)

Align to Structure: Aligning Large Language Models with Structural Information
by: Kim, Zae Myung, et al.
Published: (2025)

FedPDPO: Federated Personalized Direct Preference Optimization for Large Language Model Alignment
by: Zhu, Kewen, et al.
Published: (2026)

LongAlign: A Recipe for Long Context Alignment of Large Language Models
by: Bai, Yushi, et al.
Published: (2024)

Federated Fine-Tuning of Large Language Models: Kahneman-Tversky vs. Direct Preference Optimization
by: Spadea, Fernando, et al.
Published: (2025)

Stepwise Alignment for Constrained Language Model Policy Optimization
by: Wachi, Akifumi, et al.
Published: (2024)

AlignTune: Modular Toolkit for Post-Training Alignment of Large Language Models
by: Lyngkhoi, R E Zera Marveen, et al.
Published: (2026)

DiaTool-DPO: Multi-Turn Direct Preference Optimization for Tool-Augmented Large Language Models
by: Jung, Sunghee, et al.
Published: (2025)

Learning to Align, Aligning to Learn: A Unified Approach for Self-Optimized Alignment
by: Wang, Haowen, et al.
Published: (2025)

LifeAlign: Lifelong Alignment for Large Language Models with Memory-Augmented Focalized Preference Optimization
by: Li, Junsong, et al.
Published: (2025)

GUNDAM: Aligning Large Language Models with Graph Understanding
by: Ouyang, Sheng, et al.
Published: (2024)

Aligning Large Language Models by On-Policy Self-Judgment
by: Lee, Sangkyu, et al.
Published: (2024)

Aligning Large Language Models to Low-Resource Languages through LLM-Based Selective Translation: A Systematic Study
by: Paul, Rakesh, et al.
Published: (2025)

Efficient Post-Training Pruning of Large Language Models with Statistical Correction
by: Yu, Peiqi, et al.
Published: (2026)

Semantic Consistency Regularization with Large Language Models for Semi-supervised Sentiment Analysis
by: Li, Kunrong, et al.
Published: (2025)

To Know or Not To Know? Analyzing Self-Consistency of Large Language Models under Ambiguity
by: Sedova, Anastasiia, et al.
Published: (2024)

Comparing Bad Apples to Good Oranges: Aligning Large Language Models via Joint Preference Optimization
by: Bansal, Hritik, et al.
Published: (2024)

Federated Large Language Models: Current Progress and Future Directions
by: Yao, Yuhang, et al.
Published: (2024)

A Critical Evaluation of AI Feedback for Aligning Large Language Models
by: Sharma, Archit, et al.
Published: (2024)

A Multimodal Approach to Device-Directed Speech Detection with Large Language Models
by: Wagner, Dominik, et al.
Published: (2024)

Distilling Robustness into Natural Language Inference Models with Domain-Targeted Augmentation
by: Stacey, Joe, et al.
Published: (2023)

Did the Neurons Read your Book? Document-level Membership Inference for Large Language Models
by: Meeus, Matthieu, et al.
Published: (2023)

Cal-DPO: Calibrated Direct Preference Optimization for Language Model Alignment
by: Xiao, Teng, et al.
Published: (2024)

Found in the Middle: Permutation Self-Consistency Improves Listwise Ranking in Large Language Models
by: Tang, Raphael, et al.
Published: (2023)

Aligning Large Language Models via Fine-grained Supervision
by: Xu, Dehong, et al.
Published: (2024)

AlignBench: Benchmarking Chinese Alignment of Large Language Models
by: Liu, Xiao, et al.
Published: (2023)

The Consistency Hypothesis in Uncertainty Quantification for Large Language Models
by: Xiao, Quan, et al.
Published: (2025)

Knowledge-based Consistency Testing of Large Language Models
by: Rajan, Sai Sathiesh, et al.
Published: (2024)

Uncertainty-Aware Exploratory Direct Preference Optimization for Multimodal Large Language Models
by: Zhang, Huatian, et al.
Published: (2026)

A Survey on Medical Large Language Models: Technology, Application, Trustworthiness, and Future Directions
by: Liu, Lei, et al.
Published: (2024)

Multi-Reference Preference Optimization for Large Language Models
by: Le, Hung, et al.
Published: (2024)

Benchmarking Large Language Model Uncertainty for Prompt Optimization
by: Guo, Pei-Fu, et al.
Published: (2024)

Self-Rewarding PPO: Aligning Large Language Models with Demonstrations Only
by: Zhang, Qingru, et al.
Published: (2025)