Feedback-to-Rubrics: Can We Learn Expert Criteria from Inline Comments?
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2026
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866916061048209408 |
|---|---|
| author | Yoshida, Kotaro Kuroki, So Imajuku, Yuki Nakamura, Taishi Iwai, Ryunosuke Goda, Haruki Akiba, Takuya |
| author_facet | Yoshida, Kotaro Kuroki, So Imajuku, Yuki Nakamura, Taishi Iwai, Ryunosuke Goda, Haruki Akiba, Takuya |
| contents | Large language models (LLMs) are increasingly used for writing and review support, but their usefulness depends on context-dependent criteria, such as expert preferences or organization-specific conventions, that are often tacit, undocumented, and difficult to elicit directly. We propose a problem setting for learning reusable natural-language rubrics from accumulated inline comments on artifacts such as human-written or LLM-generated drafts. Our method infers rubrics from these comments and iteratively refines them by observing comment-wise mismatches between rubric-conditioned predictions and reference comments. We evaluate the proposed method in real-world review settings and in controlled settings with reference rubrics. These results show that inline comments can be distilled into reusable rubrics that support comment prediction, rubric understanding, and automatic artifact revision. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2605_29857 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | Feedback-to-Rubrics: Can We Learn Expert Criteria from Inline Comments? Yoshida, Kotaro Kuroki, So Imajuku, Yuki Nakamura, Taishi Iwai, Ryunosuke Goda, Haruki Akiba, Takuya Machine Learning Large language models (LLMs) are increasingly used for writing and review support, but their usefulness depends on context-dependent criteria, such as expert preferences or organization-specific conventions, that are often tacit, undocumented, and difficult to elicit directly. We propose a problem setting for learning reusable natural-language rubrics from accumulated inline comments on artifacts such as human-written or LLM-generated drafts. Our method infers rubrics from these comments and iteratively refines them by observing comment-wise mismatches between rubric-conditioned predictions and reference comments. We evaluate the proposed method in real-world review settings and in controlled settings with reference rubrics. These results show that inline comments can be distilled into reusable rubrics that support comment prediction, rubric understanding, and automatic artifact revision. |
| title | Feedback-to-Rubrics: Can We Learn Expert Criteria from Inline Comments? |
| topic | Machine Learning |
| url | https://arxiv.org/abs/2605.29857 |