Cover Image

Changing Answer Order Can Decrease MMLU Accuracy

Fuente: arXiv

Saved in:

Bibliographic Details
Main Authors:	Gupta, Vipul, Pantoja, David, Ross, Candace, Williams, Adina, Ung, Megan
Format:	Preprint
Published:	2024
Subjects:	Computation and Language
Online Access:	Acceder al recurso
Tags:	Add Tag No Tags, Be the first to tag this record!

Similar Items

Improving Model Evaluation using SMART Filtering of Benchmark Datasets
by: Gupta, Vipul, et al.
Published: (2024)

Beg to Differ: Understanding Reasoning-Answer Misalignment Across Languages
by: Ovalle, Anaelia, et al.
Published: (2025)

What makes a good metric? Evaluating automatic metrics for text-to-image consistency
by: Ross, Candace, et al.
Published: (2024)

Arbiters of Ambivalence: Challenges of Using LLMs in No-Consensus Tasks
by: Radharapu, Bhaktipriya, et al.
Published: (2025)

Chained Tuning Leads to Biased Forgetting
by: Ung, Megan, et al.
Published: (2024)

MMLU-Pro+: Evaluating Higher-Order Reasoning and Shortcut Learning in LLMs
by: Taghanaki, Saeid Asgari, et al.
Published: (2024)

Are We Done with MMLU?
by: Gema, Aryo Pradipta, et al.
Published: (2024)

Improving Text-to-Image Consistency via Automatic Prompt Optimization
by: Mañas, Oscar, et al.
Published: (2024)

[Call for Papers] The 2nd BabyLM Challenge: Sample-efficient pretraining on a developmentally plausible corpus
by: Choshen, Leshem, et al.
Published: (2024)

Can LLMs Rank the Harmfulness of Smaller LLMs? We are Not There Yet
by: Atil, Berk, et al.
Published: (2025)

Findings of the Second BabyLM Challenge: Sample-Efficient Pretraining on Developmentally Plausible Corpora
by: Hu, Michael Y., et al.
Published: (2024)

No Answer Needed: Predicting LLM Answer Accuracy from Question-Only Linear Probes
by: Cencerrado, Iván Vicente Moreno, et al.
Published: (2025)

Spanish and LLM Benchmarks: is MMLU Lost in Translation?
by: Plaza, Irene, et al.
Published: (2024)

TurkishMMLU: Measuring Massive Multitask Language Understanding in Turkish
by: Yüksel, Arda, et al.
Published: (2024)

ArabicMMLU: Assessing Massive Multitask Language Understanding in Arabic
by: Koto, Fajri, et al.
Published: (2024)

IslamicMMLU: A Benchmark for Evaluating LLMs on Islamic Knowledge
by: Abdelaal, Ali, et al.
Published: (2026)

LAG-MMLU: Benchmarking Frontier LLM Understanding in Latvian and Giriama
by: Etori, Naome A., et al.
Published: (2025)

BnMMLU: Measuring Massive Multitask Language Understanding in Bengali
by: Joy, Saman Sarker, et al.
Published: (2025)

Global MMLU: Understanding and Addressing Cultural and Linguistic Biases in Multilingual Evaluation
by: Singh, Shivalika, et al.
Published: (2024)

Are Female Carpenters like Blue Bananas? A Corpus Investigation of Occupation Gender Typicality
by: Ju, Da, et al.
Published: (2024)

Domain Regeneration: How well do LLMs match syntactic properties of text domains?
by: Ju, Da, et al.
Published: (2025)

KazMMLU: Evaluating Language Models on Kazakh, Russian, and Regional Knowledge of Kazakhstan
by: Togmanov, Mukhammed, et al.
Published: (2025)

SinhalaMMLU: A Comprehensive Benchmark for Evaluating Multitask Language Understanding in Sinhala
by: Pramodya, Ashmari, et al.
Published: (2025)

Mobile-MMLU: A Mobile Intelligence Language Understanding Benchmark
by: Bsharat, Sondos Mahmoud, et al.
Published: (2025)

Model Unlearning Objectives Vary for Distinct Language Functions
by: Atil, Berk, et al.
Published: (2026)

MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmark
by: Wang, Yubo, et al.
Published: (2024)

GreekMMLU: A Native-Sourced Multitask Benchmark for Evaluating Language Models in Greek
by: Zhang, Yang, et al.
Published: (2026)

MMLU-ProX: A Multilingual Benchmark for Advanced Large Language Model Evaluation
by: Xuan, Weihao, et al.
Published: (2025)

Rethinking Dense Sequential Chains: Reasoning Language Models Can Extract Answers from Sparse, Order-Shuffling Chain-of-Thoughts
by: Chen, Yi-Chang, et al.
Published: (2026)

What's in Common? Multimodal Models Hallucinate When Reasoning Across Scenes
by: Ross, Candace, et al.
Published: (2025)

Khayyam Challenge (PersianMMLU): Is Your LLM Truly Wise to The Persian Language?
by: Ghahroodi, Omid, et al.
Published: (2024)

Reactor Mk.1 performances: MMLU, HumanEval and BBH test results
by: Dunham, TJ, et al.
Published: (2024)

DialectalArabicMMLU: Benchmarking Dialectal Capabilities in Arabic and Multilingual Language Models
by: Altakrori, Malik H., et al.
Published: (2025)

BabyLM Turns 3: Call for papers for the 2025 BabyLM workshop
by: Charpentier, Lucas, et al.
Published: (2025)

LLMs Can Generate a Better Answer by Aggregating Their Own Responses
by: Li, Zichong, et al.
Published: (2025)

Reverse Question Answering: Can an LLM Write a Question so Hard (or Bad) that it Can't Answer?
by: Balepur, Nishant, et al.
Published: (2024)

BioACE: An Automated Framework for Biomedical Answer and Citation Evaluations
by: Gupta, Deepak, et al.
Published: (2026)

IndicMMLU-Pro: Benchmarking Indic Large Language Models on Multi-Task Language Understanding
by: KJ, Sankalp, et al.
Published: (2025)

SWE-QA: Can Language Models Answer Repository-level Code Questions?
by: Peng, Weihan, et al.
Published: (2025)

Semantic Consistency for Assuring Reliability of Large Language Models
by: Raj, Harsh, et al.
Published: (2023)