SCRuB: Social Concept Reasoning under Rubric-Based Evaluation
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2026
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866911657848995840 |
|---|---|
| author | Watson-Daniels, Jamelle Bhattacharjee, Himaghna Wang, Skyler Handoko, Brandon Li, Antonio Ovalle, Anaelia Pasupuleti, Mahesh Ross, Candace Sarma, Vidya Subramonian, Arjun Ullrich, Karen van der Vaart, Will Xin, Yijing Nickel, Maximilian |
| author_facet | Watson-Daniels, Jamelle Bhattacharjee, Himaghna Wang, Skyler Handoko, Brandon Li, Antonio Ovalle, Anaelia Pasupuleti, Mahesh Ross, Candace Sarma, Vidya Subramonian, Arjun Ullrich, Karen van der Vaart, Will Xin, Yijing Nickel, Maximilian |
| contents | While many studies of Large Language Model (LLM) reasoning capabilities emphasize mathematical or technical tasks, few address reasoning about social concepts: the abstract ideas shaping social norms, culture, and institutions. This understudied capability is essential for modern models acting as social agents, yet no systematic evaluation methodology targets it. We introduce SCRuB (Social Concept Reasoning under Rubric-Based Evaluation), a framework designed for this setting of task indeterminacy. Our goal is to measure the degree to which a model reasons about social concepts with the depth and critical rigor of a human expert. SCRuB proceeds in three phases: prompt construction from established sources, response generation by experts and models, and comparative evaluation using a five-dimensional critical thinking rubric. To enable generalization of the pipeline, we introduce a Panel of Disciplinary Perspectives ensemble validated against independent expert judges. We release SCRuBEval (n=4,711 evaluation prompts) and SCRuBAnnotations (300 expert-authored responses and 150 expert comparative judgments from 45 PhD-level scholars). Our results show that frontier models consistently outperform human experts across all five rubric dimensions. Across 1,170 pairwise comparisons, expert judges ranked a model response first in 80.8% of judgments and preferred model responses overall 74.4% of the time. Ultimately, this study provides the first expert-grounded demonstration of evaluation saturation for social concept reasoning: the single-turn exam-style format has reached its ceiling for models and humans alike. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2605_06444 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | SCRuB: Social Concept Reasoning under Rubric-Based Evaluation Watson-Daniels, Jamelle Bhattacharjee, Himaghna Wang, Skyler Handoko, Brandon Li, Antonio Ovalle, Anaelia Pasupuleti, Mahesh Ross, Candace Sarma, Vidya Subramonian, Arjun Ullrich, Karen van der Vaart, Will Xin, Yijing Nickel, Maximilian Artificial Intelligence While many studies of Large Language Model (LLM) reasoning capabilities emphasize mathematical or technical tasks, few address reasoning about social concepts: the abstract ideas shaping social norms, culture, and institutions. This understudied capability is essential for modern models acting as social agents, yet no systematic evaluation methodology targets it. We introduce SCRuB (Social Concept Reasoning under Rubric-Based Evaluation), a framework designed for this setting of task indeterminacy. Our goal is to measure the degree to which a model reasons about social concepts with the depth and critical rigor of a human expert. SCRuB proceeds in three phases: prompt construction from established sources, response generation by experts and models, and comparative evaluation using a five-dimensional critical thinking rubric. To enable generalization of the pipeline, we introduce a Panel of Disciplinary Perspectives ensemble validated against independent expert judges. We release SCRuBEval (n=4,711 evaluation prompts) and SCRuBAnnotations (300 expert-authored responses and 150 expert comparative judgments from 45 PhD-level scholars). Our results show that frontier models consistently outperform human experts across all five rubric dimensions. Across 1,170 pairwise comparisons, expert judges ranked a model response first in 80.8% of judgments and preferred model responses overall 74.4% of the time. Ultimately, this study provides the first expert-grounded demonstration of evaluation saturation for social concept reasoning: the single-turn exam-style format has reached its ceiling for models and humans alike. |
| title | SCRuB: Social Concept Reasoning under Rubric-Based Evaluation |
| topic | Artificial Intelligence |
| url | https://arxiv.org/abs/2605.06444 |