Copertina

Improving Score Reliability of Multiple Choice Benchmarks with Consistency Evaluation and Altered Answer Choices

Fuente: arXiv

Salvato in:

Dettagli Bibliografici
Autori principali:	Cavalin, Paulo, Sanctos, Cassia, Grave, Marcelo, Pinhanez, Claudio, Primerano, Yago
Natura:	Preprint
Pubblicazione:	2025
Soggetti:	Computation and Language Artificial Intelligence
Accesso online:	Acceder al recurso
Tags:	Aggiungi Tag Nessun Tag, puoi essere il primo ad aggiungerne!!

Documenti analoghi

The Non-Determinism of Small LLMs: Evidence of Low Answer Consistency in Repetition Trials of Standard Multiple-Choice Benchmarks
di: Pinhanez, Claudio, et al.
Pubblicazione: (2025)

CAT: A Metric-Driven Framework for Analyzing the Consistency-Accuracy Relation of LLMs under Controlled Input Variations
di: Cavalin, Paulo, et al.
Pubblicazione: (2025)

Exploring Performance Variations in Finetuned Translators of Ultra-Low Resource Languages: Do Linguistic Differences Matter?
di: Gonçalves, Isabel, et al.
Pubblicazione: (2025)

Answer Matching Outperforms Multiple Choice for Language Model Evaluation
di: Chandak, Nikhil, et al.
Pubblicazione: (2025)

GeoChallenge: A Multi-Answer Multiple-Choice Benchmark for Geometric Reasoning with Diagrams
di: Zhang, Yushun, et al.
Pubblicazione: (2026)

Answer, Assemble, Ace: Understanding How LMs Answer Multiple Choice Questions
di: Wiegreffe, Sarah, et al.
Pubblicazione: (2024)

Sentence-level Aggregation of Lexical Metrics Correlates Stronger with Human Judgements than Corpus-level Aggregation
di: Cavalin, Paulo, et al.
Pubblicazione: (2024)

Differentiating Choices via Commonality for Multiple-Choice Question Answering
di: Deng, Wenqing, et al.
Pubblicazione: (2024)

Plausibly Problematic Questions in Multiple-Choice Benchmarks for Commonsense Reasoning
di: Palta, Shramay, et al.
Pubblicazione: (2024)

Anchored Answers: Unravelling Positional Bias in GPT-2's Multiple-Choice Questions
di: Li, Ruizhe, et al.
Pubblicazione: (2024)

AraSTEM: A Native Arabic Multiple Choice Question Benchmark for Evaluating LLMs Knowledge In STEM Subjects
di: Mustapha, Ahmad, et al.
Pubblicazione: (2024)

Harnessing the Power of Artificial Intelligence to Vitalize Endangered Indigenous Languages: Technologies and Experiences
di: Pinhanez, Claudio, et al.
Pubblicazione: (2024)

Generating Plausible Distractors for Multiple-Choice Questions via Student Choice Prediction
di: Lee, Yooseop, et al.
Pubblicazione: (2025)

Question Difficulty Ranking for Multiple-Choice Reading Comprehension
di: Raina, Vatsal, et al.
Pubblicazione: (2024)

Exploring Iterative Enhancement for Improving Learnersourced Multiple-Choice Question Explanations with Large Language Models
di: Bao, Qiming, et al.
Pubblicazione: (2023)

Wait, that's not an option: LLMs Robustness with Incorrect Multiple-Choice Options
di: Góral, Gracjan, et al.
Pubblicazione: (2024)

STED and Consistency Scoring: A Framework for Evaluating LLM Structured Output Reliability
di: Wang, Guanghui, et al.
Pubblicazione: (2025)

Creating an African American-Sounding TTS: Guidelines, Technical Challenges,and Surprising Evaluations
di: Pinhanez, Claudio, et al.
Pubblicazione: (2024)

MCQG-SRefine: Multiple Choice Question Generation and Evaluation with Iterative Self-Critique, Correction, and Comparison Feedback
di: Yao, Zonghai, et al.
Pubblicazione: (2024)

A Vietnamese Dataset for Text Segmentation and Multiple Choices Reading Comprehension
di: Hai, Toan Nguyen, et al.
Pubblicazione: (2025)

Self-Correcting Large Language Models: Generation vs. Multiple Choice
di: Rahmani, Hossein A., et al.
Pubblicazione: (2025)

Automated Generation and Tagging of Knowledge Components from Multiple-Choice Questions
di: Moore, Steven, et al.
Pubblicazione: (2024)

SAS-Bench: A Fine-Grained Benchmark for Evaluating Short Answer Scoring with Large Language Models
di: Lai, Peichao, et al.
Pubblicazione: (2025)

Option-ID Based Elimination For Multiple Choice Questions
di: Zhu, Zhenhao, et al.
Pubblicazione: (2025)

Biomedical Entity Linking as Multiple Choice Question Answering
di: Lin, Zhenxi, et al.
Pubblicazione: (2024)

Latent Self-Consistency for Reliable Majority-Set Selection in Short- and Long-Answer Reasoning
di: Oh, Jungsuk, et al.
Pubblicazione: (2025)

Beyond Multiple Choice: Verifiable OpenQA for Robust Vision-Language RFT
di: Liu, Yesheng, et al.
Pubblicazione: (2025)

Boosting Process-Correct CoT Reasoning by Modeling Solvability of Multiple-Choice QA
di: Schumann, Raphael, et al.
Pubblicazione: (2025)

From Multiple-Choice to Extractive QA: A Case Study for English and Arabic
di: Lynn, Teresa, et al.
Pubblicazione: (2024)

SATA-BENCH: Select All That Apply Benchmark for Multiple Choice Questions
di: Xu, Weijie, et al.
Pubblicazione: (2025)

Multiple-Choice Question Generation Using Large Language Models: Methodology and Educator Insights
di: Biancini, Giorgio, et al.
Pubblicazione: (2025)

Conformal P-Value in Multiple-Choice Question Answering Tasks with Provable Risk Control
di: Ye, Yuanchang
Pubblicazione: (2025)

Multiple Choice Learning of Low-Rank Adapters for Language Modeling
di: Letzelter, Victor, et al.
Pubblicazione: (2025)

Balancing Rigor and Utility: Mitigating Cognitive Biases in Large Language Models for Multiple-Choice Questions
di: Zhong, Hanyang, et al.
Pubblicazione: (2024)

Choices Speak Louder than Questions
di: Cho, Gyeongje, et al.
Pubblicazione: (2025)

Right Answer, Wrong Score: Uncovering the Inconsistencies of LLM Evaluation in Multiple-Choice Question Answering
di: Molfese, Francesco Maria, et al.
Pubblicazione: (2025)

Pattern Recognition or Medical Knowledge? The Problem with Multiple-Choice Questions in Medicine
di: Griot, Maxime, et al.
Pubblicazione: (2024)

NLP at UC Santa Cruz at SemEval-2024 Task 5: Legal Answer Validation using Few-Shot Multi-Choice QA
di: Pahilajani, Anish, et al.
Pubblicazione: (2024)

It is Too Many Options: Pitfalls of Multiple-Choice Questions in Generative AI and Medical Education
di: Singh, Shrutika, et al.
Pubblicazione: (2025)

PerCoR: Evaluating Commonsense Reasoning in Persian via Multiple-Choice Sentence Completion
di: Alikhani, Morteza, et al.
Pubblicazione: (2025)