The Pursuit of Empathy: Evaluating Small Language Models for PTSD Dialogue Support
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2025
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866908548819058688 |
|---|---|
| author | BN, Suhas Mahajan, Yash Mattioli, Dominik Sherrill, Andrew M. Arriaga, Rosa I. Wiese, Chris W. Abdullah, Saeed |
| author_facet | BN, Suhas Mahajan, Yash Mattioli, Dominik Sherrill, Andrew M. Arriaga, Rosa I. Wiese, Chris W. Abdullah, Saeed |
| contents | This paper investigates the capacity of small language models (0.5B-5B parameters) to generate empathetic responses for individuals with PTSD. We introduce Trauma-Informed Dialogue for Empathy (TIDE), a novel dataset comprising 10,000 two-turn conversations across 500 diverse, clinically-grounded PTSD personas (https://huggingface.co/datasets/yenopoya/TIDE). Using frontier model outputs as ground truth, we evaluate eight small LLMs in zero-shot settings and after fine-tuning. Fine-tuning enhances empathetic capabilities, improving cosine similarity and perceived empathy, although gains vary across emotional scenarios and smaller models exhibit a "knowledge transfer ceiling." As expected, Claude Sonnet 3.5 consistently outperforms all models, but surprisingly, the smaller models often approach human-rated empathy levels. Demographic analyses showed that older adults favored responses that validated distress before offering support (p = .004), while graduate-educated users preferred emotionally layered replies in specific scenarios. Gender-based differences were minimal (p > 0.15), suggesting the feasibility of broadly empathetic model designs. This work offers insights into building resource-efficient, emotionally intelligent systems for mental health support. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2505_15065 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | The Pursuit of Empathy: Evaluating Small Language Models for PTSD Dialogue Support BN, Suhas Mahajan, Yash Mattioli, Dominik Sherrill, Andrew M. Arriaga, Rosa I. Wiese, Chris W. Abdullah, Saeed Computation and Language Artificial Intelligence Computers and Society 68T50, 68T05 I.2.7; I.2.1; H.5.2 This paper investigates the capacity of small language models (0.5B-5B parameters) to generate empathetic responses for individuals with PTSD. We introduce Trauma-Informed Dialogue for Empathy (TIDE), a novel dataset comprising 10,000 two-turn conversations across 500 diverse, clinically-grounded PTSD personas (https://huggingface.co/datasets/yenopoya/TIDE). Using frontier model outputs as ground truth, we evaluate eight small LLMs in zero-shot settings and after fine-tuning. Fine-tuning enhances empathetic capabilities, improving cosine similarity and perceived empathy, although gains vary across emotional scenarios and smaller models exhibit a "knowledge transfer ceiling." As expected, Claude Sonnet 3.5 consistently outperforms all models, but surprisingly, the smaller models often approach human-rated empathy levels. Demographic analyses showed that older adults favored responses that validated distress before offering support (p = .004), while graduate-educated users preferred emotionally layered replies in specific scenarios. Gender-based differences were minimal (p > 0.15), suggesting the feasibility of broadly empathetic model designs. This work offers insights into building resource-efficient, emotionally intelligent systems for mental health support. |
| title | The Pursuit of Empathy: Evaluating Small Language Models for PTSD Dialogue Support |
| topic | Computation and Language Artificial Intelligence Computers and Society 68T50, 68T05 I.2.7; I.2.1; H.5.2 |
| url | https://arxiv.org/abs/2505.15065 |