RL-Struct: A Lightweight Reinforcement Learning Framework for Reliable Structured Output in LLMs
Fuente:
arXiv
Saved in:
| Main Authors: | , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866909961998565376 |
|---|---|
| author | Hu, Ruike Wu, Shulei |
| author_facet | Hu, Ruike Wu, Shulei |
| contents | The Structure Gap between probabilistic LLM generation and deterministic schema requirements hinders automated workflows. We propose RL-Struct, a lightweight framework using Gradient Regularized Policy Optimization (GRPO) with a hierarchical reward function to align LLMs with structural constraints. This approach eliminates the critic network, reducing peak VRAM by 38% compared to PPO. On complex JSON tasks, RL-Struct achieves 89.7% structural accuracy and 92.1% validity, significantly outperforming SFT and zero-shot baselines. We also report an emergent curriculum--a self-organized learning process where the model prioritizes syntax before semantics. Our model is publicly available at https://huggingface.co/Freakz3z/Qwen-JSON. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2512_00319 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | RL-Struct: A Lightweight Reinforcement Learning Framework for Reliable Structured Output in LLMs Hu, Ruike Wu, Shulei Artificial Intelligence Machine Learning The Structure Gap between probabilistic LLM generation and deterministic schema requirements hinders automated workflows. We propose RL-Struct, a lightweight framework using Gradient Regularized Policy Optimization (GRPO) with a hierarchical reward function to align LLMs with structural constraints. This approach eliminates the critic network, reducing peak VRAM by 38% compared to PPO. On complex JSON tasks, RL-Struct achieves 89.7% structural accuracy and 92.1% validity, significantly outperforming SFT and zero-shot baselines. We also report an emergent curriculum--a self-organized learning process where the model prioritizes syntax before semantics. Our model is publicly available at https://huggingface.co/Freakz3z/Qwen-JSON. |
| title | RL-Struct: A Lightweight Reinforcement Learning Framework for Reliable Structured Output in LLMs |
| topic | Artificial Intelligence Machine Learning |
| url | https://arxiv.org/abs/2512.00319 |