Toward Systematic Counterfactual Fairness Evaluation of Large Language Models: The CAFFE Framework
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866918255382233088 |
|---|---|
| author | Parziale, Alessandra Voria, Gianmario Pontillo, Valeria Catolino, Gemma De Lucia, Andrea Palomba, Fabio |
| author_facet | Parziale, Alessandra Voria, Gianmario Pontillo, Valeria Catolino, Gemma De Lucia, Andrea Palomba, Fabio |
| contents | Nowadays, Large Language Models (LLMs) are foundational components of modern software systems. As their influence grows, concerns about fairness have become increasingly pressing. Prior work has proposed metamorphic testing to detect fairness issues, applying input transformations to uncover inconsistencies in model behavior. This paper introduces an alternative perspective for testing counterfactual fairness in LLMs, proposing a structured and intent-aware framework coined CAFFE (Counterfactual Assessment Framework for Fairness Evaluation). Inspired by traditional non-functional testing, CAFFE (1) formalizes LLM-Fairness test cases through explicitly defined components, including prompt intent, conversational context, input variants, expected fairness thresholds, and test environment configuration, (2) assists testers by automatically generating targeted test data, and (3) evaluates model responses using semantic similarity metrics. Our experiments, conducted on three different architectural families of LLM, demonstrate that CAFFE achieves broader bias coverage and more reliable detection of unfair behavior than existing metamorphic approaches. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2512_16816 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Toward Systematic Counterfactual Fairness Evaluation of Large Language Models: The CAFFE Framework Parziale, Alessandra Voria, Gianmario Pontillo, Valeria Catolino, Gemma De Lucia, Andrea Palomba, Fabio Software Engineering Nowadays, Large Language Models (LLMs) are foundational components of modern software systems. As their influence grows, concerns about fairness have become increasingly pressing. Prior work has proposed metamorphic testing to detect fairness issues, applying input transformations to uncover inconsistencies in model behavior. This paper introduces an alternative perspective for testing counterfactual fairness in LLMs, proposing a structured and intent-aware framework coined CAFFE (Counterfactual Assessment Framework for Fairness Evaluation). Inspired by traditional non-functional testing, CAFFE (1) formalizes LLM-Fairness test cases through explicitly defined components, including prompt intent, conversational context, input variants, expected fairness thresholds, and test environment configuration, (2) assists testers by automatically generating targeted test data, and (3) evaluates model responses using semantic similarity metrics. Our experiments, conducted on three different architectural families of LLM, demonstrate that CAFFE achieves broader bias coverage and more reliable detection of unfair behavior than existing metamorphic approaches. |
| title | Toward Systematic Counterfactual Fairness Evaluation of Large Language Models: The CAFFE Framework |
| topic | Software Engineering |
| url | https://arxiv.org/abs/2512.16816 |