FairEval: Evaluating Fairness in LLM-Based Recommendations with Personality Awareness
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2025
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866916683793301504 |
|---|---|
| author | Sah, Chandan Kumar Lian, Xiaoli Xu, Tony Zhang, Li |
| author_facet | Sah, Chandan Kumar Lian, Xiaoli Xu, Tony Zhang, Li |
| contents | Recent advances in Large Language Models (LLMs) have enabled their application to recommender systems (RecLLMs), yet concerns remain regarding fairness across demographic and psychological user dimensions. We introduce FairEval, a novel evaluation framework to systematically assess fairness in LLM-based recommendations. FairEval integrates personality traits with eight sensitive demographic attributes,including gender, race, and age, enabling a comprehensive assessment of user-level bias. We evaluate models, including ChatGPT 4o and Gemini 1.5 Flash, on music and movie recommendations. FairEval's fairness metric, PAFS, achieves scores up to 0.9969 for ChatGPT 4o and 0.9997 for Gemini 1.5 Flash, with disparities reaching 34.79 percent. These results highlight the importance of robustness in prompt sensitivity and support more inclusive recommendation systems. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2504_07801 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | FairEval: Evaluating Fairness in LLM-Based Recommendations with Personality Awareness Sah, Chandan Kumar Lian, Xiaoli Xu, Tony Zhang, Li Information Retrieval Artificial Intelligence Human-Computer Interaction Recent advances in Large Language Models (LLMs) have enabled their application to recommender systems (RecLLMs), yet concerns remain regarding fairness across demographic and psychological user dimensions. We introduce FairEval, a novel evaluation framework to systematically assess fairness in LLM-based recommendations. FairEval integrates personality traits with eight sensitive demographic attributes,including gender, race, and age, enabling a comprehensive assessment of user-level bias. We evaluate models, including ChatGPT 4o and Gemini 1.5 Flash, on music and movie recommendations. FairEval's fairness metric, PAFS, achieves scores up to 0.9969 for ChatGPT 4o and 0.9997 for Gemini 1.5 Flash, with disparities reaching 34.79 percent. These results highlight the importance of robustness in prompt sensitivity and support more inclusive recommendation systems. |
| title | FairEval: Evaluating Fairness in LLM-Based Recommendations with Personality Awareness |
| topic | Information Retrieval Artificial Intelligence Human-Computer Interaction |
| url | https://arxiv.org/abs/2504.07801 |