Imagen de Portada

Which of These Best Describes Multiple Choice Evaluation with LLMs? A) Forced B) Flawed C) Fixable D) All of the Above

Fuente: arXiv

Guardado en:

Detalles Bibliográficos
Autores principales:	Balepur, Nishant, Rudinger, Rachel, Boyd-Graber, Jordan Lee
Formato:	Preprint
Publicado:	2025
Materias:	Computation and Language
Acceso en línea:	Acceder al recurso
Etiquetas:	Agregar Etiqueta Sin Etiquetas, Sea el primero en etiquetar este registro!

Ejemplares similares

BenchMarker: An Education-Inspired Toolkit for Highlighting Flaws in Multiple-Choice Benchmarks
por: Balepur, Nishant, et al.
Publicado: (2026)

Artifacts or Abduction: How Do LLMs Answer Multiple-Choice Questions Without the Question?
por: Balepur, Nishant, et al.
Publicado: (2024)

Is Your Large Language Model Knowledgeable or a Choices-Only Cheater?
por: Balepur, Nishant, et al.
Publicado: (2024)

Test-Time Reasoners Are Strategic Multiple-Choice Test-Takers
por: Balepur, Nishant, et al.
Publicado: (2025)

Reverse Question Answering: Can an LLM Write a Question so Hard (or Bad) that it Can't Answer?
por: Balepur, Nishant, et al.
Publicado: (2024)

Whose Boat Does it Float? Improving Personalization in Preference Tuning via Inferred User Personas
por: Balepur, Nishant, et al.
Publicado: (2025)

DiscoTrace: Representing and Comparing Answering Strategies of Humans and LLMs in Information-Seeking Question Answering
por: Srikanth, Neha, et al.
Publicado: (2026)

It's Not Easy Being Wrong: Large Language Models Struggle with Process of Elimination Reasoning
por: Balepur, Nishant, et al.
Publicado: (2023)

Plausibly Problematic Questions in Multiple-Choice Benchmarks for Commonsense Reasoning
por: Palta, Shramay, et al.
Publicado: (2024)

KARL: Knowledge-Aware Retrieval and Representations aid Retention and Learning in Students
por: Shu, Matthew, et al.
Publicado: (2024)

A Good Plan is Hard to Find: Aligning Models with Preferences is Misaligned with What Helps Users
por: Balepur, Nishant, et al.
Publicado: (2025)

A SMART Mnemonic Sounds like "Glue Tonic": Mixing LLMs with Student Feedback to Make Mnemonic Learning Stick
por: Balepur, Nishant, et al.
Publicado: (2024)

DRACULA: Hunting for the Actions Users Want Deep Research Agents to Execute
por: Balepur, Nishant, et al.
Publicado: (2026)

MODS: Moderating a Mixture of Document Speakers to Summarize Debatable Queries in Document Collections
por: Balepur, Nishant, et al.
Publicado: (2025)

Language Models Don't Know What You Want: Evaluating Personalization in Deep Research Needs Real Users
por: Balepur, Nishant, et al.
Publicado: (2026)

Pregnant Questions: The Importance of Pragmatic Awareness in Maternal Health Question Answering
por: Srikanth, Neha, et al.
Publicado: (2023)

Large Language Models Struggle to Describe the Haystack without Human Help: Human-in-the-loop Evaluation of Topic Models
por: Li, Zongxia, et al.
Publicado: (2025)

Can They Dixit? Yes they Can! Dixit as a Playground for Multimodal Language Model Capabilities
por: Balepur, Nishant, et al.
Publicado: (2025)

ProxAnn: Use-Oriented Evaluations of Topic Models and Document Clustering
por: Hoyle, Alexander, et al.
Publicado: (2025)

Is your benchmark truly adversarial? AdvScore: Evaluating Human-Grounded Adversarialness
por: Sung, Yoo Yeon, et al.
Publicado: (2024)

GRACE: A Granular Benchmark for Evaluating Model Calibration against Human Calibration
por: Sung, Yoo Yeon, et al.
Publicado: (2025)

CFMatch: Aligning Automated Answer Equivalence Evaluation with Expert Judgments For Open-Domain Question Answering
por: Li, Zongxia, et al.
Publicado: (2024)

How the Advent of Ubiquitous Large Language Models both Stymie and Turbocharge Dynamic Adversarial Question Generation
por: Sung, Yoo Yeon, et al.
Publicado: (2024)

Mirror, Mirror on the Wall -- Which is the Best Model of Them All?
por: Sayed, Dina, et al.
Publicado: (2025)

Labeled Interactive Topic Models
por: Seelman, Kyle, et al.
Publicado: (2023)

NLI under the Microscope: What Atomic Hypothesis Decomposition Reveals
por: Srikanth, Neha, et al.
Publicado: (2025)

Multiple LLM Agents Debate for Equitable Cultural Alignment
por: Ki, Dayeon, et al.
Publicado: (2025)

NAVIG: Natural Language-guided Analysis with Vision Language Models for Image Geo-localization
por: Zhang, Zheyuan, et al.
Publicado: (2025)

Susu Box or Piggy Bank: Assessing Cultural Commonsense Knowledge between Ghana and the U.S
por: Acquaye, Christabel, et al.
Publicado: (2024)

On the Influence of Gender and Race in Romantic Relationship Prediction from Large Language Models
por: Sancheti, Abhilasha, et al.
Publicado: (2024)

SMART-Editor: A Multi-Agent Framework for Human-Like Design Editing with Structural Integrity
por: Mondal, Ishani, et al.
Publicado: (2025)

PEDANTS: Cheap but Effective and Interpretable Answer Equivalence
por: Li, Zongxia, et al.
Publicado: (2024)

None of the Above, Less of the Right: Parallel Patterns between Humans and LLMs on Multi-Choice Questions Answering
por: Tam, Zhi Rui, et al.
Publicado: (2025)

Brittle Minds, Fixable Activations: Understanding Belief Representations in Language Models
por: Bortoletto, Matteo, et al.
Publicado: (2024)

How often are errors in natural language reasoning due to paraphrastic variability?
por: Srikanth, Neha, et al.
Publicado: (2024)

Flaw or Artifact? Rethinking Prompt Sensitivity in Evaluating LLMs
por: Hua, Andong, et al.
Publicado: (2025)

Evaluating LLMs with Multiple Problems at once
por: Wang, Zhengxiang, et al.
Publicado: (2024)

Pushing the Boundaries of Multiple Choice Evaluation to One Hundred Options
por: Lee, Nahyun, et al.
Publicado: (2026)

AUDITA: A New Dataset to Audit Humans vs. AI Skill at Audio QA
por: Kabir, Tasnim, et al.
Publicado: (2026)

Do great minds think alike? Investigating Human-AI Complementarity in Question Answering with CAIMIRA
por: Gor, Maharshi, et al.
Publicado: (2024)