Imagen de Portada

It's Not You, It's Clipping: A Soft Trust-Region via Probability Smoothing for LLM RL

Fuente: arXiv

Guardado en:

Detalles Bibliográficos
Autores principales:	Dwyer, Madeleine, Sobey, Adam, Chapman, Adriane
Formato:	Preprint
Publicado:	2025
Materias:	Machine Learning Artificial Intelligence
Acceso en línea:	Acceder al recurso
Etiquetas:	Agregar Etiqueta Sin Etiquetas, Sea el primero en etiquetar este registro!

Ejemplares similares

BandPO: Bridging Trust Regions and Ratio Clipping via Probability-Aware Bounds for LLM Reinforcement Learning
por: Li, Yuan, et al.
Publicado: (2026)

From $\log π$ to $π$: Taming Divergence in Soft Clipping via Bilateral Decoupled Decay of Probability Gradient Weight
por: Fu, Xiaoliang, et al.
Publicado: (2026)

Graph-attention-based Casual Discovery with Trust Region-navigated Clipping Policy Optimization
por: Liu, Shixuan, et al.
Publicado: (2024)

SAC-GLAM: Improving Online RL for LLM agents with Soft Actor-Critic and Hindsight Relabeling
por: Gaven, Loris, et al.
Publicado: (2024)

Clip Your Sequences Fairly: Enforcing Length Fairness for Sequence-Level RL
por: Mao, Hanyi, et al.
Publicado: (2025)

Soft Deterministic Policy Gradient with Gaussian Smoothing
por: Na, Hyunjun, et al.
Publicado: (2026)

PPO-Clip Attains Global Optimality: Towards Deeper Understandings of Clipping
por: Huang, Nai-Chieh, et al.
Publicado: (2023)

What Can You Do When You Have Zero Rewards During RL?
por: Prakash, Jatin, et al.
Publicado: (2025)

Smooth Gate Functions for Soft Advantage Policy Optimization
por: Denisov, Egor, et al.
Publicado: (2026)

Trust Region Preference Approximation: A simple and stable reinforcement learning algorithm for LLM reasoning
por: Su, Xuerui, et al.
Publicado: (2025)

Trust Regions for Explanations via Black-Box Probabilistic Certification
por: Dhurandhar, Amit, et al.
Publicado: (2024)

Offline RL with Smooth OOD Generalization in Convex Hull and its Neighborhood
por: Yao, Qingmao, et al.
Publicado: (2025)

Trust Regions Sell, But Who's Buying? Overlap Geometry as an Alternative Trust Region for Policy Optimization
por: Trivedi, Gaurish, et al.
Publicado: (2026)

Trust Region Masking for Long-Horizon LLM Reinforcement Learning
por: Li, Yingru, et al.
Publicado: (2025)

Trust the Batch, On- or Off-Policy: Adaptive Policy Optimization for RL Post-Training
por: Fakoor, Rasool, et al.
Publicado: (2026)

Trust-Region Adaptive Policy Optimization
por: Su, Mingyu, et al.
Publicado: (2025)

Soft Policy Optimization: Online Off-Policy RL for Sequence Models
por: Cohen, Taco, et al.
Publicado: (2025)

On Entropy Control in LLM-RL Algorithms
por: Shen, Han
Publicado: (2025)

Token-Efficient RL for LLM Reasoning
por: Lee, Alan, et al.
Publicado: (2025)

Revisiting Training Scale: An Empirical Study of Token Count, Power Consumption, and Parameter Efficiency
por: Dwyer, Joe
Publicado: (2026)

Understanding Fixed Predictions via Confined Regions
por: Lawless, Connor, et al.
Publicado: (2025)

Plan Before You Trade: Inference-Time Optimization for RL Trading Agents
por: Go, Eun, et al.
Publicado: (2026)

CluCERT: Certifying LLM Robustness via Clustering-Guided Denoising Smoothing
por: Wang, Zixia, et al.
Publicado: (2025)

Trust-Region Behavior Blending for On-Policy Distillation
por: Plyusov, Daniil, et al.
Publicado: (2026)

Effective Confidence Region Prediction Using Probability Forecasters
por: Lindsay, David, et al.
Publicado: (2024)

Attention Smoothing Is All You Need For Unlearning
por: Zade, Saleh Zare, et al.
Publicado: (2026)

Combining LLM decision and RL action selection to improve RL policy for adaptive interventions
por: Karine, Karine, et al.
Publicado: (2025)

It's About Time: Temporal References in Emergent Communication
por: Lipinski, Olaf, et al.
Publicado: (2023)

RL$^3$: Boosting Meta Reinforcement Learning via RL inside RL$^2$
por: Bhatia, Abhinav, et al.
Publicado: (2023)

Weight Clipping for Deep Continual and Reinforcement Learning
por: Elsayed, Mohamed, et al.
Publicado: (2024)

HELENE: Hessian Layer-wise Clipping and Gradient Annealing for Accelerating Fine-tuning LLM with Zeroth-order Optimization
por: Zhao, Huaqin, et al.
Publicado: (2024)

Can You Break RLVER? Probing Adversarial Robustness of RL-Trained Empathetic Agents
por: K, Deeraj S, et al.
Publicado: (2026)

Don't flatten, tokenize! Unlocking the key to SoftMoE's efficacy in deep RL
por: Sokar, Ghada, et al.
Publicado: (2024)

Can You Trust an LLM with Your Life-Changing Decision? An Investigation into AI High-Stakes Responses
por: Cahyono, Joshua Adrian, et al.
Publicado: (2025)

Matrix Low-Rank Trust Region Policy Optimization
por: Rozada, Sergio, et al.
Publicado: (2024)

Towards Realistic Guarantees: A Probabilistic Certificate for SmoothLLM
por: Kumarappan, Adarsh, et al.
Publicado: (2025)

Trust Region Q Adjoint Matching
por: Dong, Yonghoon, et al.
Publicado: (2026)

Do Not Let Low-Probability Tokens Over-Dominate in RL for LLMs
por: Yang, Zhihe, et al.
Publicado: (2025)

Accelerating RL for LLM Reasoning with Optimal Advantage Regression
por: Brantley, Kianté, et al.
Publicado: (2025)

You Need Reasoning to Learn Reasoning: The Limitations of Label-Free RL in Weak Base Models
por: Roy, Shuvendu, et al.
Publicado: (2025)