Cover Image

Nested-ReFT: Efficient Reinforcement Learning for Large Language Model Fine-Tuning via Off-Policy Rollouts

Fuente: arXiv

Saved in:

Bibliographic Details
Main Authors:	Heuillet, Maxime, Cui, Yufei, Chen, Boxing, Durand, Audrey, Parthasarathi, Prasanna
Format:	Preprint
Published:	2025
Subjects:	Machine Learning Artificial Intelligence Computation and Language
Online Access:	Acceder al recurso
Tags:	Add Tag No Tags, Be the first to tag this record!

Similar Items

ReFT: Reasoning with Reinforced Fine-Tuning
by: Luong, Trung Quoc, et al.
Published: (2024)

ReFT: Representation Finetuning for Language Models
by: Wu, Zhengxuan, et al.
Published: (2024)

Thinking Long, but Short: Stable Sequential Test-Time Scaling for Large Reasoning Models
by: Metel, Michael R., et al.
Published: (2026)

Beyond Mode-Seeking RL: Trajectory-Balance Post-Training for Diffusion Language Models
by: Ahmadi, Saba, et al.
Published: (2026)

Neural Active Learning Meets the Partial Monitoring Framework
by: Heuillet, Maxime, et al.
Published: (2024)

LLM-as-a-Judge: Toward World Models for Slate Recommendation Systems
by: Bonin, Baptiste, et al.
Published: (2025)

Randomized Confidence Bounds for Stochastic Partial Monitoring
by: Heuillet, Maxime, et al.
Published: (2024)

Resona: Improving Context Copying in Linear Recurrence Models with Retrieval
by: Wang, Xinyu, et al.
Published: (2025)

Do Robot Snakes Dream like Electric Sheep? Investigating the Effects of Architectural Inductive Biases on Hallucination
by: Huang, Jerry, et al.
Published: (2024)

GRPO-$λ$: Credit Assignment improves LLM Reasoning
by: Parthasarathi, Prasanna, et al.
Published: (2025)

CHARP: Conversation History AwaReness Probing for Knowledge-grounded Dialogue Systems
by: Ghaddar, Abbas, et al.
Published: (2024)

One-Token Rollout: Guiding Supervised Fine-Tuning of LLMs with Policy Gradient
by: Ming, Rui, et al.
Published: (2025)

Robust Fine-Tuning from Non-Robust Pretrained Models: Mitigating Suboptimal Transfer With Epsilon-Scheduling
by: Ngnawé, Jonas, et al.
Published: (2025)

S2D: Sorted Speculative Decoding For More Efficient Deployment of Nested Large Language Models
by: Kavehzadeh, Parsa, et al.
Published: (2024)

LiFT: Does Instruction Fine-Tuning Improve In-Context Learning for Longitudinal Modelling by Large Language Models?
by: Ali, Iqra, et al.
Published: (2026)

Do Large Language Models Know How Much They Know?
by: Prato, Gabriele, et al.
Published: (2025)

Off-Policy Value-Based Reinforcement Learning for Large Language Models
by: Wang, Peng-Yuan, et al.
Published: (2026)

ChunkFT: Byte-Streamed Optimization for Memory-Efficient Full Fine-Tuning
by: Liu, Yongkang, et al.
Published: (2026)

BOSCH: Black-Box Binary Optimization for Short-Context Attention-Head Selection in LLMs
by: Ghaddar, Abbas, et al.
Published: (2026)

A General Highly Accurate Online Planning Method Integrating Large Language Models into Nested Rollout Policy Adaptation for Dialogue Tasks
by: Wang, Hui, et al.
Published: (2025)

Mitigating Outlier Activations in Low-Precision Fine-Tuning of Language Models
by: Ghaffari, Alireza, et al.
Published: (2023)

SPEC-RL: Accelerating On-Policy Reinforcement Learning with Speculative Rollouts
by: Liu, Bingshuai, et al.
Published: (2025)

InfMem: Learning System-2 Memory Control for Long-Context Agent
by: Wang, Xinyu, et al.
Published: (2026)

Towards Practical Tool Usage for Continually Learning LLMs
by: Huang, Jerry, et al.
Published: (2024)

Knowledgeable Agents by Offline Reinforcement Learning from Large Language Model Rollouts
by: Pang, Jing-Cheng, et al.
Published: (2024)

GiFT: Gibbs Fine-Tuning for Code Generation
by: Li, Haochen, et al.
Published: (2025)

Efficient Differentially Private Fine-Tuning of LLMs via Reinforcement Learning
by: Khadangi, Afshin, et al.
Published: (2025)

Not All Rollouts are Useful: Down-Sampling Rollouts in LLM Reinforcement Learning
by: Xu, Yixuan Even, et al.
Published: (2025)

HierRouter: Coordinated Routing of Specialized Large Language Models via Reinforcement Learning
by: Gupta, Nikunj, et al.
Published: (2025)

HiFT: A Hierarchical Full Parameter Fine-Tuning Strategy
by: Liu, Yongkang, et al.
Published: (2024)

Mind the Gap: Data Rewriting for Stable Off-Policy Supervised Fine-Tuning
by: Zhao, Shiwan, et al.
Published: (2025)

Parameter-Efficient Fine-Tuning of Large Language Models via Deconvolution in Subspace
by: Zhang, Jia-Chen, et al.
Published: (2025)

Large Language Models for Anomaly Detection in Computational Workflows: from Supervised Fine-Tuning to In-Context Learning
by: Jin, Hongwei, et al.
Published: (2024)

Fine-Tuning Language Models with Reward Learning on Policy
by: Lang, Hao, et al.
Published: (2024)

Parameter-Efficient Fine-Tuning of Large Language Models using Semantic Knowledge Tuning
by: Prottasha, Nusrat Jahan, et al.
Published: (2024)

QDyLoRA: Quantized Dynamic Low-Rank Adaptation for Efficient Large Language Model Tuning
by: Rajabzadeh, Hossein, et al.
Published: (2024)

SEM: Reinforcement Learning for Search-Efficient Large Language Models
by: Sha, Zeyang, et al.
Published: (2025)

Re-Ranking Step by Step: Investigating Pre-Filtering for Re-Ranking with Large Language Models
by: Nouriinanloo, Baharan, et al.
Published: (2024)

Fine-Tuning Large Vision-Language Models as Decision-Making Agents via Reinforcement Learning
by: Zhai, Yuexiang, et al.
Published: (2024)

Are Full Rollouts Necessary for On-Policy Distillation?
by: Zhang, Yaocheng, et al.
Published: (2026)