SAGE: Scalable Automated Robustness Augmentation for LLM Knowledge Evaluation
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2026
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866917484999737344 |
|---|---|
| author | Li, Xiaoyuan Wang, Yuzhe Li, Moxin Bao, Keqin Men, Rui Zhang, Yichang Liu, Dayiheng Wang, Wenjie Feng, Fuli |
| author_facet | Li, Xiaoyuan Wang, Yuzhe Li, Moxin Bao, Keqin Men, Rui Zhang, Yichang Liu, Dayiheng Wang, Wenjie Feng, Fuli |
| contents | Large Language Models (LLMs) achieve strong performance on standard knowledge evaluation benchmarks, yet recent work shows that their knowledge capabilities remain brittle under question variants that test the same knowledge in different forms. Robustness augmentation of existing knowledge evaluation benchmarks is therefore necessary, but current LLM-assisted generate-then-verify pipelines are costly and difficult to scale due to low-yield variant generation and unreliable variant verification. We propose SAGE (Scalable Automated Generation of Robustness BEnchmarks), a framework for scalable robustness augmentation of knowledge evaluation benchmarks using fine-tuned smaller models. SAGE consists of VariantQual, a rubric-based verifier trained on human-labeled seed data, and VariantGen, a variant generator initialized with supervised fine-tuning and further optimized with reinforcement learning using VariantQual as the reward model. Experiments on HellaSwag show that SAGE constructs a large-scale robustness-augmented benchmark with quality comparable to the human-annotated HellaSwag-Pro at substantially lower cost, while the fine-tuned models further generalize to MMLU without benchmark-specific fine-tuning. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2605_12022 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | SAGE: Scalable Automated Robustness Augmentation for LLM Knowledge Evaluation Li, Xiaoyuan Wang, Yuzhe Li, Moxin Bao, Keqin Men, Rui Zhang, Yichang Liu, Dayiheng Wang, Wenjie Feng, Fuli Computation and Language Large Language Models (LLMs) achieve strong performance on standard knowledge evaluation benchmarks, yet recent work shows that their knowledge capabilities remain brittle under question variants that test the same knowledge in different forms. Robustness augmentation of existing knowledge evaluation benchmarks is therefore necessary, but current LLM-assisted generate-then-verify pipelines are costly and difficult to scale due to low-yield variant generation and unreliable variant verification. We propose SAGE (Scalable Automated Generation of Robustness BEnchmarks), a framework for scalable robustness augmentation of knowledge evaluation benchmarks using fine-tuned smaller models. SAGE consists of VariantQual, a rubric-based verifier trained on human-labeled seed data, and VariantGen, a variant generator initialized with supervised fine-tuning and further optimized with reinforcement learning using VariantQual as the reward model. Experiments on HellaSwag show that SAGE constructs a large-scale robustness-augmented benchmark with quality comparable to the human-annotated HellaSwag-Pro at substantially lower cost, while the fine-tuned models further generalize to MMLU without benchmark-specific fine-tuning. |
| title | SAGE: Scalable Automated Robustness Augmentation for LLM Knowledge Evaluation |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2605.12022 |