Social Bias Benchmark for Generation: A Comparison of Generation and QA-Based Evaluations
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2025
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866908405476622336 |
|---|---|
| author | Jin, Jiho Kang, Woosung Myung, Junho Oh, Alice |
| author_facet | Jin, Jiho Kang, Woosung Myung, Junho Oh, Alice |
| contents | Measuring social bias in large language models (LLMs) is crucial, but existing bias evaluation methods struggle to assess bias in long-form generation. We propose a Bias Benchmark for Generation (BBG), an adaptation of the Bias Benchmark for QA (BBQ), designed to evaluate social bias in long-form generation by having LLMs generate continuations of story prompts. Building our benchmark in English and Korean, we measure the probability of neutral and biased generations across ten LLMs. We also compare our long-form story generation evaluation results with multiple-choice BBQ evaluation, showing that the two approaches produce inconsistent results. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2503_06987 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Social Bias Benchmark for Generation: A Comparison of Generation and QA-Based Evaluations Jin, Jiho Kang, Woosung Myung, Junho Oh, Alice Computation and Language Artificial Intelligence Measuring social bias in large language models (LLMs) is crucial, but existing bias evaluation methods struggle to assess bias in long-form generation. We propose a Bias Benchmark for Generation (BBG), an adaptation of the Bias Benchmark for QA (BBQ), designed to evaluate social bias in long-form generation by having LLMs generate continuations of story prompts. Building our benchmark in English and Korean, we measure the probability of neutral and biased generations across ten LLMs. We also compare our long-form story generation evaluation results with multiple-choice BBQ evaluation, showing that the two approaches produce inconsistent results. |
| title | Social Bias Benchmark for Generation: A Comparison of Generation and QA-Based Evaluations |
| topic | Computation and Language Artificial Intelligence |
| url | https://arxiv.org/abs/2503.06987 |