Gespeichert in:
| Hauptverfasser: | , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2025
|
| Schlagworte: | |
| Online-Zugang: | https://arxiv.org/abs/2510.20782 |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866912666660896768 |
|---|---|
| author | Sagae, Alicia Lee, Chia-Jung Avula, Sandeep Dang, Brandon Murdock, Vanessa |
| author_facet | Sagae, Alicia Lee, Chia-Jung Avula, Sandeep Dang, Brandon Murdock, Vanessa |
| contents | Current methods for evaluating large language models (LLMs) typically focus on high-level tasks such as text generation, without targeting a particular AI application. This approach is not sufficient for evaluating LLMs for Responsible AI dimensions like fairness, since protected attributes that are highly relevant in one application may be less relevant in another. In this work, we construct a dataset that is driven by a real-world application (generate a plain-text product description, given a list of product features), parameterized by fairness attributes intersected with gendered adjectives and product categories, yielding a rich set of labeled prompts. We show how to use the data to identify quality, veracity, safety, and fairness gaps in LLMs, contributing a proposal for LLM evaluation paired with a concrete resource for the research community. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2510_20782 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | A Use-Case Specific Dataset for Measuring Dimensions of Responsible Performance in LLM-generated Text Sagae, Alicia Lee, Chia-Jung Avula, Sandeep Dang, Brandon Murdock, Vanessa Computation and Language Artificial Intelligence I.2.7 Current methods for evaluating large language models (LLMs) typically focus on high-level tasks such as text generation, without targeting a particular AI application. This approach is not sufficient for evaluating LLMs for Responsible AI dimensions like fairness, since protected attributes that are highly relevant in one application may be less relevant in another. In this work, we construct a dataset that is driven by a real-world application (generate a plain-text product description, given a list of product features), parameterized by fairness attributes intersected with gendered adjectives and product categories, yielding a rich set of labeled prompts. We show how to use the data to identify quality, veracity, safety, and fairness gaps in LLMs, contributing a proposal for LLM evaluation paired with a concrete resource for the research community. |
| title | A Use-Case Specific Dataset for Measuring Dimensions of Responsible Performance in LLM-generated Text |
| topic | Computation and Language Artificial Intelligence I.2.7 |
| url | https://arxiv.org/abs/2510.20782 |