A Japanese Benchmark for Evaluating Social Bias in Reasoning Based on Attribution Theory
Fuente:
arXiv
Salvato in:
| Autori principali: | , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2026
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866908930402156544 |
|---|---|
| author | Shiotani, Taihei Kaneko, Masahiro Okazaki, Naoaki |
| author_facet | Shiotani, Taihei Kaneko, Masahiro Okazaki, Naoaki |
| contents | In enhancing the fairness of Large Language Models (LLMs), evaluating social biases rooted in the cultural contexts of specific linguistic regions is essential. However, most existing Japanese benchmarks heavily rely on translating English data, which does not necessarily provide an evaluation suitable for Japanese culture. Furthermore, they only evaluate bias in the conclusion, failing to capture biases lurking in the reasoning. In this study, based on attribution theory in social psychology, we constructed a new dataset, ``JUBAKU-v2,'' which evaluates the bias in attributing behaviors to in-groups and out-groups within reasoning while fixing the conclusion. This dataset consists of 216 examples reflecting cultural biases specific to Japan. Experimental results verified that it can detect performance differences across models more sensitively than existing benchmarks. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2604_00568 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | A Japanese Benchmark for Evaluating Social Bias in Reasoning Based on Attribution Theory Shiotani, Taihei Kaneko, Masahiro Okazaki, Naoaki Computation and Language In enhancing the fairness of Large Language Models (LLMs), evaluating social biases rooted in the cultural contexts of specific linguistic regions is essential. However, most existing Japanese benchmarks heavily rely on translating English data, which does not necessarily provide an evaluation suitable for Japanese culture. Furthermore, they only evaluate bias in the conclusion, failing to capture biases lurking in the reasoning. In this study, based on attribution theory in social psychology, we constructed a new dataset, ``JUBAKU-v2,'' which evaluates the bias in attributing behaviors to in-groups and out-groups within reasoning while fixing the conclusion. This dataset consists of 216 examples reflecting cultural biases specific to Japan. Experimental results verified that it can detect performance differences across models more sensitively than existing benchmarks. |
| title | A Japanese Benchmark for Evaluating Social Bias in Reasoning Based on Attribution Theory |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2604.00568 |