LFQA-HP-1M: A Large-Scale Human Preference Dataset for Long-Form Question Answering
Fuente:
arXiv
Saved in:
| Main Authors: | , , |
|---|---|
| Format: | Preprint |
| Published: |
2026
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866917298962432000 |
|---|---|
| author | Jahan, Rafid Ishrak Iqbal, Fahmid Shahriar Choudhury, Sagnik Ray |
| author_facet | Jahan, Rafid Ishrak Iqbal, Fahmid Shahriar Choudhury, Sagnik Ray |
| contents | Long-form question answering (LFQA) demands nuanced evaluation of multi-sentence explanatory responses, yet existing metrics often fail to reflect human judgment. We present LFQA-HP-1M, a large-scale dataset comprising 1.3M human pairwise preference annotations for LFQA. We propose nine rubrics for answer quality evaluation, and show that simple linear models based on these features perform comparably to state-of-the-art LLM evaluators. We further examine transitivity consistency, positional bias, and verbosity biases in LLM evaluators and demonstrate their vulnerability to adversarial perturbations. Overall, this work provides one of the largest public LFQA preference datasets and a rubric-driven framework for transparent and reliable evaluation. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2602_23603 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | LFQA-HP-1M: A Large-Scale Human Preference Dataset for Long-Form Question Answering Jahan, Rafid Ishrak Iqbal, Fahmid Shahriar Choudhury, Sagnik Ray Computation and Language Artificial Intelligence Information Retrieval Long-form question answering (LFQA) demands nuanced evaluation of multi-sentence explanatory responses, yet existing metrics often fail to reflect human judgment. We present LFQA-HP-1M, a large-scale dataset comprising 1.3M human pairwise preference annotations for LFQA. We propose nine rubrics for answer quality evaluation, and show that simple linear models based on these features perform comparably to state-of-the-art LLM evaluators. We further examine transitivity consistency, positional bias, and verbosity biases in LLM evaluators and demonstrate their vulnerability to adversarial perturbations. Overall, this work provides one of the largest public LFQA preference datasets and a rubric-driven framework for transparent and reliable evaluation. |
| title | LFQA-HP-1M: A Large-Scale Human Preference Dataset for Long-Form Question Answering |
| topic | Computation and Language Artificial Intelligence Information Retrieval |
| url | https://arxiv.org/abs/2602.23603 |