JBBQ: Japanese Bias Benchmark for Analyzing Social Biases in Large Language Models
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2024
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866915340928155648 |
|---|---|
| author | Yanaka, Hitomi Han, Namgi Kumon, Ryoma Lu, Jie Takeshita, Masashi Sekizawa, Ryo Kato, Taisei Arai, Hiromi |
| author_facet | Yanaka, Hitomi Han, Namgi Kumon, Ryoma Lu, Jie Takeshita, Masashi Sekizawa, Ryo Kato, Taisei Arai, Hiromi |
| contents | With the development of large language models (LLMs), social biases in these LLMs have become a pressing issue. Although there are various benchmarks for social biases across languages, the extent to which Japanese LLMs exhibit social biases has not been fully investigated. In this study, we construct the Japanese Bias Benchmark dataset for Question Answering (JBBQ) based on the English bias benchmark BBQ, with analysis of social biases in Japanese LLMs. The results show that while current open Japanese LLMs with more parameters show improved accuracies on JBBQ, their bias scores increase. In addition, prompts with a warning about social biases and chain-of-thought prompting reduce the effect of biases in model outputs, but there is room for improvement in extracting the correct evidence from contexts in Japanese. Our dataset is available at https://github.com/ynklab/JBBQ_data. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2406_02050 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | JBBQ: Japanese Bias Benchmark for Analyzing Social Biases in Large Language Models Yanaka, Hitomi Han, Namgi Kumon, Ryoma Lu, Jie Takeshita, Masashi Sekizawa, Ryo Kato, Taisei Arai, Hiromi Computation and Language With the development of large language models (LLMs), social biases in these LLMs have become a pressing issue. Although there are various benchmarks for social biases across languages, the extent to which Japanese LLMs exhibit social biases has not been fully investigated. In this study, we construct the Japanese Bias Benchmark dataset for Question Answering (JBBQ) based on the English bias benchmark BBQ, with analysis of social biases in Japanese LLMs. The results show that while current open Japanese LLMs with more parameters show improved accuracies on JBBQ, their bias scores increase. In addition, prompts with a warning about social biases and chain-of-thought prompting reduce the effect of biases in model outputs, but there is room for improvement in extracting the correct evidence from contexts in Japanese. Our dataset is available at https://github.com/ynklab/JBBQ_data. |
| title | JBBQ: Japanese Bias Benchmark for Analyzing Social Biases in Large Language Models |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2406.02050 |