ImplicitBBQ: Benchmarking Implicit Bias in Large Language Models through Characteristic Based Cues
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2026
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866912998597066752 |
|---|---|
| author | Vedula, Bhaskara Hanuma Anghan, Darshan Goyal, Ishita Kumaraguru, Ponnurangam Chakraborty, Abhijnan |
| author_facet | Vedula, Bhaskara Hanuma Anghan, Darshan Goyal, Ishita Kumaraguru, Ponnurangam Chakraborty, Abhijnan |
| contents | Large Language Models increasingly suppress biased outputs when demographic identity is stated explicitly, yet may still exhibit implicit biases when identity is conveyed indirectly. Existing benchmarks use name based proxies to detect implicit biases, which carry weak associations with many social demographics and cannot extend to dimensions like age or socioeconomic status. We introduce ImplicitBBQ, a QA benchmark that evaluates implicit bias through characteristic based cues, culturally associated attributes that signal implicitly, across age, gender, region, religion, caste, and socioeconomic status. Evaluating 11 models, we find that implicit bias in ambiguous contexts is over six times higher than explicit bias in open weight models. Safety prompting and chain-of-thought reasoning fail to substantially close this gap; even few-shot prompting, which reduces implicit bias by 84%, leaves caste bias at four times the level of any other dimension. These findings indicate that current alignment and prompting strategies address the surface of bias evaluation while leaving culturally grounded stereotypic associations largely unresolved. We publicly release our code and dataset for model providers and researchers to benchmark potential mitigation techniques. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2604_01925 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | ImplicitBBQ: Benchmarking Implicit Bias in Large Language Models through Characteristic Based Cues Vedula, Bhaskara Hanuma Anghan, Darshan Goyal, Ishita Kumaraguru, Ponnurangam Chakraborty, Abhijnan Computation and Language Artificial Intelligence Large Language Models increasingly suppress biased outputs when demographic identity is stated explicitly, yet may still exhibit implicit biases when identity is conveyed indirectly. Existing benchmarks use name based proxies to detect implicit biases, which carry weak associations with many social demographics and cannot extend to dimensions like age or socioeconomic status. We introduce ImplicitBBQ, a QA benchmark that evaluates implicit bias through characteristic based cues, culturally associated attributes that signal implicitly, across age, gender, region, religion, caste, and socioeconomic status. Evaluating 11 models, we find that implicit bias in ambiguous contexts is over six times higher than explicit bias in open weight models. Safety prompting and chain-of-thought reasoning fail to substantially close this gap; even few-shot prompting, which reduces implicit bias by 84%, leaves caste bias at four times the level of any other dimension. These findings indicate that current alignment and prompting strategies address the surface of bias evaluation while leaving culturally grounded stereotypic associations largely unresolved. We publicly release our code and dataset for model providers and researchers to benchmark potential mitigation techniques. |
| title | ImplicitBBQ: Benchmarking Implicit Bias in Large Language Models through Characteristic Based Cues |
| topic | Computation and Language Artificial Intelligence |
| url | https://arxiv.org/abs/2604.01925 |