XChoice: Explainable Evaluation of AI-Human Alignment in LLM-based Constrained Choice Decision Making
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2026
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866911380020396032 |
|---|---|
| author | Qi, Weihong Huang, Fan Muralidharan, Rasika An, Jisun Kwak, Haewoon |
| author_facet | Qi, Weihong Huang, Fan Muralidharan, Rasika An, Jisun Kwak, Haewoon |
| contents | We present XChoice, an explainable framework for evaluating AI-human alignment in constrained decision making. Moving beyond outcome agreement such as accuracy and F1 score, XChoice fits a mechanism-based decision model to human data and LLM-generated decisions, recovering interpretable parameters that capture the relative importance of decision factors, constraint sensitivity, and implied trade-offs. Alignment is assessed by comparing these parameter vectors across models, options, and subgroups. We demonstrate XChoice on Americans' daily time allocation using the American Time Use Survey (ATUS) as human ground truth, revealing heterogeneous alignment across models and activities and salient misalignment concentrated in Black and married groups. We further validate robustness of XChoice via an invariance analysis and evaluate targeted mitigation with a retrieval augmented generation (RAG) intervention. Overall, XChoice provides mechanism-based metrics that diagnose misalignment and support informed improvements beyond surface outcome matching. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2601_11286 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | XChoice: Explainable Evaluation of AI-Human Alignment in LLM-based Constrained Choice Decision Making Qi, Weihong Huang, Fan Muralidharan, Rasika An, Jisun Kwak, Haewoon Artificial Intelligence We present XChoice, an explainable framework for evaluating AI-human alignment in constrained decision making. Moving beyond outcome agreement such as accuracy and F1 score, XChoice fits a mechanism-based decision model to human data and LLM-generated decisions, recovering interpretable parameters that capture the relative importance of decision factors, constraint sensitivity, and implied trade-offs. Alignment is assessed by comparing these parameter vectors across models, options, and subgroups. We demonstrate XChoice on Americans' daily time allocation using the American Time Use Survey (ATUS) as human ground truth, revealing heterogeneous alignment across models and activities and salient misalignment concentrated in Black and married groups. We further validate robustness of XChoice via an invariance analysis and evaluate targeted mitigation with a retrieval augmented generation (RAG) intervention. Overall, XChoice provides mechanism-based metrics that diagnose misalignment and support informed improvements beyond surface outcome matching. |
| title | XChoice: Explainable Evaluation of AI-Human Alignment in LLM-based Constrained Choice Decision Making |
| topic | Artificial Intelligence |
| url | https://arxiv.org/abs/2601.11286 |