Saved in:
| Main Authors: | , , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2511.04070 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866918507294228480 |
|---|---|
| author | Havaldar, Shreya You, Weiqiu Kim, Chaehyeon Xue, Anton Jin, Helen Gatti, Marco Jain, Bhuvnesh Qu, Helen Madani, Amin Hashimoto, Daniel A. Weissman, Gary E. Deo, Rajat Khatana, Sameed Ungar, Lyle Wong, Eric |
| author_facet | Havaldar, Shreya You, Weiqiu Kim, Chaehyeon Xue, Anton Jin, Helen Gatti, Marco Jain, Bhuvnesh Qu, Helen Madani, Amin Hashimoto, Daniel A. Weissman, Gary E. Deo, Rajat Khatana, Sameed Ungar, Lyle Wong, Eric |
| contents | As LLMs are deployed in knowledge-intensive settings (e.g., surgery, astronomy, therapy), users are often domain experts who expect not just answers, but explanations that mirror professional reasoning. Yet evaluating whether an LLM "thinks like an expert" remains difficult: existing approaches rely on per-example expert annotation, making them costly, hard to scale, and tied to a single notion of correct reasoning within each domain. To address this gap, we introduce T-FIX, a unified evaluation framework that operationalizes expert alignment as a desired attribute of LLM-generated explanations. T-FIX spans seven scientific tasks across three domains, with each task evaluated against expert-defined criteria that capture domain-grounded reasoning rather than generic explanation quality. Our framework enables automatic, personalizable evaluation of expert alignment that generalizes to unseen explanations without ongoing expert involvement. Code is available at https://github.com/BrachioLab/FIX-2/. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2511_04070 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | T-FIX: Text-Based Explanations with Features Interpretable to eXperts Havaldar, Shreya You, Weiqiu Kim, Chaehyeon Xue, Anton Jin, Helen Gatti, Marco Jain, Bhuvnesh Qu, Helen Madani, Amin Hashimoto, Daniel A. Weissman, Gary E. Deo, Rajat Khatana, Sameed Ungar, Lyle Wong, Eric Computation and Language As LLMs are deployed in knowledge-intensive settings (e.g., surgery, astronomy, therapy), users are often domain experts who expect not just answers, but explanations that mirror professional reasoning. Yet evaluating whether an LLM "thinks like an expert" remains difficult: existing approaches rely on per-example expert annotation, making them costly, hard to scale, and tied to a single notion of correct reasoning within each domain. To address this gap, we introduce T-FIX, a unified evaluation framework that operationalizes expert alignment as a desired attribute of LLM-generated explanations. T-FIX spans seven scientific tasks across three domains, with each task evaluated against expert-defined criteria that capture domain-grounded reasoning rather than generic explanation quality. Our framework enables automatic, personalizable evaluation of expert alignment that generalizes to unseen explanations without ongoing expert involvement. Code is available at https://github.com/BrachioLab/FIX-2/. |
| title | T-FIX: Text-Based Explanations with Features Interpretable to eXperts |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2511.04070 |