Image de couverture de livre

A Framework for Evaluating LLMs Under Task Indeterminacy

Fuente: arXiv

Enregistré dans:

Détails bibliographiques
Auteurs principaux:	Guerdan, Luke, Wallach, Hanna, Barocas, Solon, Chouldechova, Alexandra
Format:	Preprint
Publié:	2024
Sujets:	Machine Learning Computation and Language Human-Computer Interaction
Accès en ligne:	Acceder al recurso
Tags:	Ajouter un tag Pas de tags, Soyez le premier à ajouter un tag!

Documents similaires

Validating LLM-as-a-Judge Systems under Rating Indeterminacy
par: Guerdan, Luke, et autres
Publié: (2025)

Effects of Generative AI Errors on User Reliance Across Task Difficulty
par: Anthis, Jacy Reese, et autres
Publié: (2026)

Measurement as Bricolage: Examining How Data Scientists Construct Target Variables for Predictive Modeling Tasks
par: Guerdan, Luke, et autres
Publié: (2025)

Leveraging Expert Consistency to Improve Algorithmic Decision Support
par: De-Arteaga, Maria, et autres
Publié: (2021)

"One-Size-Fits-All"? Examining Expectations around What Constitute "Fair" or "Good" NLG System Behaviors
par: Lucy, Li, et autres
Publié: (2023)

Toward Cultural Interpretability: A Linguistic Anthropological Framework for Describing and Evaluating Large Language Models (LLMs)
par: Jones, Graham M., et autres
Publié: (2024)

Evaluation of LLMs-based Hidden States as Author Representations for Psychological Human-Centered NLP Tasks
par: Soni, Nikita, et autres
Publié: (2025)

AIRepr: An Analyst-Inspector Framework for Evaluating Reproducibility of LLMs in Data Science
par: Zeng, Qiuhai, et autres
Publié: (2025)

LLMs for XAI: Future Directions for Explaining Explanations
par: Zytek, Alexandra, et autres
Publié: (2024)

Creative Loss: Ambiguity, Uncertainty and Indeterminacy
par: Holberton, Tom
Publié: (2024)

Understanding the LLM-ification of CHI: Unpacking the Impact of LLMs at CHI through a Systematic Literature Review
par: Pang, Rock Yuren, et autres
Publié: (2025)

Evaluating the Usability of LLMs in Threat Intelligence Enrichment
par: Srikanth, Sanchana, et autres
Publié: (2024)

A Reproducible Framework for Neural Topic Modeling in Focus Group Analysis
par: Arfaoui, Heger, et autres
Publié: (2025)

Supporting Industry Computing Researchers in Assessing, Articulating, and Addressing the Potential Negative Societal Impact of Their Work
par: Deng, Wesley Hanwen, et autres
Publié: (2024)

PREF: Reference-Free Evaluation of Personalised Text Generation in LLMs
par: Fu, Xiao, et autres
Publié: (2025)

The Impossibility of Fair LLMs
par: Anthis, Jacy, et autres
Publié: (2024)

Easy Dataset: A Unified and Extensible Framework for Synthesizing LLM Fine-Tuning Data from Unstructured Documents
par: Miao, Ziyang, et autres
Publié: (2025)

The Behavior Gap: Evaluating Zero-shot LLM Agents in Complex Task-Oriented Dialogs
par: Baidya, Avinash, et autres
Publié: (2025)

Sample-Efficient Human Evaluation of Large Language Models via Maximum Discrepancy Competition
par: Feng, Kehua, et autres
Publié: (2024)

Can Generative AI Support Patients' & Caregivers' Informational Needs? Towards Task-Centric Evaluation Of AI Systems
par: Rajagopal, Shreya, et autres
Publié: (2024)

Humanizing LLMs: A Survey of Psychological Measurements with Tools, Datasets, and Human-Agent Applications
par: Dong, Wenhan, et autres
Publié: (2025)

Beyond Semantic Similarity: A Component-Wise Evaluation Framework for Medical Question Answering Systems with Health Equity Implications
par: Sakib, Abu Noman Md, et autres
Publié: (2026)

Harmonic LLMs are Trustworthy
par: Kersting, Nicholas S., et autres
Publié: (2024)

Evaluating Explanations: An Explanatory Virtues Framework for Mechanistic Interpretability -- The Strange Science Part I.ii
par: Ayonrinde, Kola, et autres
Publié: (2025)

K-QA: A Real-World Medical Q&A Benchmark
par: Manes, Itay, et autres
Publié: (2024)

Personalized Benchmarking: Evaluating LLMs by Individual Preferences
par: Garbacea, Cristina, et autres
Publié: (2026)

Interaction Dynamics as a Reward Signal for LLMs
par: Gooding, Sian, et autres
Publié: (2025)

Generative UI: LLMs are Effective UI Generators
par: Leviathan, Yaniv, et autres
Publié: (2026)

Multimodal Fusion with LLMs for Engagement Prediction in Natural Conversation
par: Ma, Cheng Charles, et autres
Publié: (2024)

Self-reflecting Large Language Models: A Hegelian Dialectical Approach
par: Abdali, Sara, et autres
Publié: (2025)

Think Like a Person Before Responding: A Multi-Faceted Evaluation of Persona-Guided LLMs for Countering Hate
par: Ngueajio, Mikel K., et autres
Publié: (2025)

JailbreakHunter: A Visual Analytics Approach for Jailbreak Prompts Discovery from Large-Scale Human-LLM Conversational Datasets
par: Jin, Zhihua, et autres
Publié: (2024)

CURE: Cultural Understanding and Reasoning Evaluation - A Framework for "Thick" Culture Alignment Evaluation in LLMs
par: Vo, Truong, et autres
Publié: (2025)

Model-in-the-Loop (MILO): Accelerating Multimodal AI Data Annotation with LLMs
par: Wang, Yifan, et autres
Publié: (2024)

Are You Being Tracked? Discover the Power of Zero-Shot Trajectory Tracing with LLMs!
par: Yang, Huanqi, et autres
Publié: (2024)

Probing the Multi-turn Planning Capabilities of LLMs via 20 Question Games
par: Zhang, Yizhe, et autres
Publié: (2023)

Perils of Label Indeterminacy: A Case Study on Prediction of Neurological Recovery After Cardiac Arrest
par: Schoeffer, Jakob, et autres
Publié: (2025)

OMNIGUARD: An Efficient Approach for AI Safety Moderation Across Languages and Modalities
par: Verma, Sahil, et autres
Publié: (2025)

Demo: Statistically Significant Results On Biases and Errors of LLMs Do Not Guarantee Generalizable Results
par: Liu, Jonathan, et autres
Publié: (2025)

Leveraging Variation Theory in Counterfactual Data Augmentation for Optimized Active Learning
par: Gebreegziabher, Simret Araya, et autres
Publié: (2024)