Salvato in:
| Autori principali: | , , , , , , , , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2026
|
| Soggetti: | |
| Accesso online: | https://arxiv.org/abs/2605.26872 |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866916049551622144 |
|---|---|
| author | Hu, Zhengyu Xiao, Zheyuan Song, Linxin Jiang, Fengqing Li, Yutai Chen, Zhengyu Xiong, Zhihan Liu, Yue Lin, Junhao Su, Yao Hu, Lijie Ding, Kaize Teng, Xiao Poovendran, Radha |
| author_facet | Hu, Zhengyu Xiao, Zheyuan Song, Linxin Jiang, Fengqing Li, Yutai Chen, Zhengyu Xiong, Zhihan Liu, Yue Lin, Junhao Su, Yao Hu, Lijie Ding, Kaize Teng, Xiao Poovendran, Radha |
| contents | LLM training increasingly relies on teacher-generated supervision, from synthetic responses to reasoning traces and tool-use demonstrations. Current practice often chooses the highest-performing teacher to generate student training data, implicitly treating teacher test performance as a proxy for teaching quality. We show that this assumption can fail: even when multiple teachers provide correct answers to the same question, the answer from the strongest teacher is not necessarily the best supervision for a given student. To address this gap, we propose Student-Centric Answer Sampling (SCAS), a framework that selects from verified teacher-generated answers according to their estimated student-centric learning cost. Motivated by a token-wise gradient decomposition, we derive an efficient forward-only proxy for this cost and use it to guide answer selection during training. Experiments across 30 teacher models, 6 student base models, and 8 tasks show that SCAS consistently improves student performance, suggesting that effective distillation should prioritize supervision matched to the current student rather than teacher strength alone. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2605_26872 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | The Strongest Teacher Is Not Always the Best Teacher: Student-Centric Answer Selection Hu, Zhengyu Xiao, Zheyuan Song, Linxin Jiang, Fengqing Li, Yutai Chen, Zhengyu Xiong, Zhihan Liu, Yue Lin, Junhao Su, Yao Hu, Lijie Ding, Kaize Teng, Xiao Poovendran, Radha Machine Learning Artificial Intelligence Computation and Language LLM training increasingly relies on teacher-generated supervision, from synthetic responses to reasoning traces and tool-use demonstrations. Current practice often chooses the highest-performing teacher to generate student training data, implicitly treating teacher test performance as a proxy for teaching quality. We show that this assumption can fail: even when multiple teachers provide correct answers to the same question, the answer from the strongest teacher is not necessarily the best supervision for a given student. To address this gap, we propose Student-Centric Answer Sampling (SCAS), a framework that selects from verified teacher-generated answers according to their estimated student-centric learning cost. Motivated by a token-wise gradient decomposition, we derive an efficient forward-only proxy for this cost and use it to guide answer selection during training. Experiments across 30 teacher models, 6 student base models, and 8 tasks show that SCAS consistently improves student performance, suggesting that effective distillation should prioritize supervision matched to the current student rather than teacher strength alone. |
| title | The Strongest Teacher Is Not Always the Best Teacher: Student-Centric Answer Selection |
| topic | Machine Learning Artificial Intelligence Computation and Language |
| url | https://arxiv.org/abs/2605.26872 |