Cover Image

Segmental Advantage Estimation: Enhancing PPO for Long-Context LLM Training

Fuente: arXiv

Saved in:

Bibliographic Details
Main Authors:	Gong, Xue, Yi, Qi, Nan, Ziyuan, Huang, Guanhua, Li, Kejiao, Jiang, Yuhao, Xiong, Ruibin, Xu, Zenan, Guo, Jiaming, Peng, Shaohui, Zhou, Bo
Format:	Preprint
Published:	2026
Subjects:	Machine Learning Artificial Intelligence
Online Access:	Acceder al recurso
Tags:	Add Tag No Tags, Be the first to tag this record!

Similar Items

Low-probability Tokens Sustain Exploration in Reinforcement Learning with Verifiable Reward
by: Huang, Guanhua, et al.
Published: (2025)

Adaptive Termination for Multi-round Parallel Reasoning: An Universal Semantic Entropy-Guided Framework
by: Xu, Zenan, et al.
Published: (2025)

Turn-PPO: Turn-Level Advantage Estimation with PPO for Improved Multi-Turn RL in Agentic LLMs
by: Li, Junbo, et al.
Published: (2025)

ArtifactsBench: Bridging the Visual-Interactive Gap in LLM Code Generation Evaluation
by: Zhang, Chenchen, et al.
Published: (2025)

SPPO: Sequence-Level PPO for Long-Horizon Reasoning Tasks
by: Wang, Tianyi, et al.
Published: (2026)

Reinforcement Learning on Pre-Training Data
by: Li, Siheng, et al.
Published: (2025)

AM-PPO: (Advantage) Alpha-Modulation with Proximal Policy Optimization
by: Sane, Soham
Published: (2025)

Seismic Performance Parameter Analysis of Prefabricated Segmental Hybrid‐Connected Double‐Column Piers
by: Zuoqiao You, et al.
Published: (2025)

Kernelized Advantage Estimation: From Nonparametric Statistics to LLM Reasoning
by: Gong, Shijin, et al.
Published: (2026)

Ex3: Automatic Novel Writing by Extracting, Excelsior and Expanding
by: Huang, Lei, et al.
Published: (2024)

Incentivizing In-depth Reasoning over Long Contexts with Process Advantage Shaping
by: Peng, Miao, et al.
Published: (2026)

StructCoh: Structured Contrastive Learning for Context-Aware Text Semantic Matching
by: Xue, Chao, et al.
Published: (2025)

Enhancing Uncertainty Estimation in LLMs with Expectation of Aggregated Internal Belief
by: Xiao, Zeguan, et al.
Published: (2025)

Advantage of utilizing nonlocal magic resource in Haar-random circuits
by: Huang, Xiao, et al.
Published: (2025)

Seismic Performance Study of Prefabricated Segmental Assembled Hybrid Connected Double‐Column Bridge Piers
by: Zuoqiao You, et al.
Published: (2025)

Learning from the Right Rollouts: Data Attribution for PPO-based LLM Post-Training
by: Shu, Dong, et al.
Published: (2026)

Beyond Length: Quantifying Long-Range Information for Long-Context LLM Pretraining Data
by: Deng, Haoran, et al.
Published: (2025)

Training-Inference Consistent Segmented Execution for Long-Context LLMs
by: Shang, Xianpeng, et al.
Published: (2026)

BASIS: Batchwise Advantage Estimation from Single-Rollout Information Sharing for LLM Reasoning
by: Gong, Shijin, et al.
Published: (2026)

Mode-Dependent Rectification for Stable PPO Training
by: Mohamad, Mohamad, et al.
Published: (2026)

Is DPO Superior to PPO for LLM Alignment? A Comprehensive Study
by: Xu, Shusheng, et al.
Published: (2024)

Quest: Query-Aware Sparsity for Efficient Long-Context LLM Inference
by: Tang, Jiaming, et al.
Published: (2024)

Efficient Long-Context LLM Inference via KV Cache Clustering
by: Hu, Jie, et al.
Published: (2025)

Long Context Pre-Training with Lighthouse Attention
by: Peng, Bowen, et al.
Published: (2026)

3D-RPE: Enhancing Long-Context Modeling Through 3D Rotary Position Encoding
by: Ma, Xindian, et al.
Published: (2024)

VLA Model Post-Training via Action-Chunked PPO and Self Behavior Cloning
by: Wang, Si-Cheng, et al.
Published: (2025)

A Self-Training Approach for Whisper to Enhance Long Dysarthric Speech Recognition
by: Wang, Shiyao, et al.
Published: (2025)

Structured Packing in LLM Training Improves Long Context Utilization
by: Staniszewski, Konrad, et al.
Published: (2023)

Quantile Advantage Estimation: Stabilizing RLVR for LLM Reasoning
by: Wu, Junkang, et al.
Published: (2025)

Confidence Interval Construction for Causally Generalized Estimates With Target Sample Summary Information
by: Yi Chen, et al.
Published: (2026)

Locret: Enhancing Eviction in Long-Context LLM Inference with Trained Retaining Heads on Consumer-Grade Devices
by: Huang, Yuxiang, et al.
Published: (2024)

LoongTrain: Efficient Training of Long-Sequence LLMs with Head-Context Parallelism
by: Gu, Diandian, et al.
Published: (2024)

InfLLM: Training-Free Long-Context Extrapolation for LLMs with an Efficient Context Memory
by: Xiao, Chaojun, et al.
Published: (2024)

Enhancing PPO with Trajectory-Aware Hybrid Policies
by: Liu, Qisai, et al.
Published: (2025)

Adaptive Alpha Weighting with PPO: Enhancing Prompt-Based LLM-Generated Alphas in Quant Trading
by: Chen, Qizhao, et al.
Published: (2025)

A Little Goes a Long Way: Efficient Long Context Training and Inference with Partial Contexts
by: Ge, Suyu, et al.
Published: (2024)

DuoAttention: Efficient Long-Context LLM Inference with Retrieval and Streaming Heads
by: Xiao, Guangxuan, et al.
Published: (2024)

Beyond the Surface: Enhancing LLM-as-a-Judge Alignment with Human via Internal Representations
by: Lai, Peng, et al.
Published: (2025)

Training-Free Long-Context Scaling of Large Language Models
by: An, Chenxin, et al.
Published: (2024)

VinePPO: Refining Credit Assignment in RL Training of LLMs
by: Kazemnejad, Amirhossein, et al.
Published: (2024)