Copertina

Changing Answer Order Can Decrease MMLU Accuracy

Fuente: arXiv

Salvato in:

Dettagli Bibliografici
Autori principali:	Gupta, Vipul, Pantoja, David, Ross, Candace, Williams, Adina, Ung, Megan
Natura:	Preprint
Pubblicazione:	2024
Soggetti:	Computation and Language
Accesso online:	Acceder al recurso
Tags:	Aggiungi Tag Nessun Tag, puoi essere il primo ad aggiungerne!!

Documenti analoghi

Improving Model Evaluation using SMART Filtering of Benchmark Datasets
di: Gupta, Vipul, et al.
Pubblicazione: (2024)

Beg to Differ: Understanding Reasoning-Answer Misalignment Across Languages
di: Ovalle, Anaelia, et al.
Pubblicazione: (2025)

What makes a good metric? Evaluating automatic metrics for text-to-image consistency
di: Ross, Candace, et al.
Pubblicazione: (2024)

Arbiters of Ambivalence: Challenges of Using LLMs in No-Consensus Tasks
di: Radharapu, Bhaktipriya, et al.
Pubblicazione: (2025)

Chained Tuning Leads to Biased Forgetting
di: Ung, Megan, et al.
Pubblicazione: (2024)

MMLU-Pro+: Evaluating Higher-Order Reasoning and Shortcut Learning in LLMs
di: Taghanaki, Saeid Asgari, et al.
Pubblicazione: (2024)

Are We Done with MMLU?
di: Gema, Aryo Pradipta, et al.
Pubblicazione: (2024)

Improving Text-to-Image Consistency via Automatic Prompt Optimization
di: Mañas, Oscar, et al.
Pubblicazione: (2024)

[Call for Papers] The 2nd BabyLM Challenge: Sample-efficient pretraining on a developmentally plausible corpus
di: Choshen, Leshem, et al.
Pubblicazione: (2024)

Can LLMs Rank the Harmfulness of Smaller LLMs? We are Not There Yet
di: Atil, Berk, et al.
Pubblicazione: (2025)

Findings of the Second BabyLM Challenge: Sample-Efficient Pretraining on Developmentally Plausible Corpora
di: Hu, Michael Y., et al.
Pubblicazione: (2024)

No Answer Needed: Predicting LLM Answer Accuracy from Question-Only Linear Probes
di: Cencerrado, Iván Vicente Moreno, et al.
Pubblicazione: (2025)

Spanish and LLM Benchmarks: is MMLU Lost in Translation?
di: Plaza, Irene, et al.
Pubblicazione: (2024)

TurkishMMLU: Measuring Massive Multitask Language Understanding in Turkish
di: Yüksel, Arda, et al.
Pubblicazione: (2024)

ArabicMMLU: Assessing Massive Multitask Language Understanding in Arabic
di: Koto, Fajri, et al.
Pubblicazione: (2024)

IslamicMMLU: A Benchmark for Evaluating LLMs on Islamic Knowledge
di: Abdelaal, Ali, et al.
Pubblicazione: (2026)

LAG-MMLU: Benchmarking Frontier LLM Understanding in Latvian and Giriama
di: Etori, Naome A., et al.
Pubblicazione: (2025)

BnMMLU: Measuring Massive Multitask Language Understanding in Bengali
di: Joy, Saman Sarker, et al.
Pubblicazione: (2025)

Global MMLU: Understanding and Addressing Cultural and Linguistic Biases in Multilingual Evaluation
di: Singh, Shivalika, et al.
Pubblicazione: (2024)

Are Female Carpenters like Blue Bananas? A Corpus Investigation of Occupation Gender Typicality
di: Ju, Da, et al.
Pubblicazione: (2024)

Domain Regeneration: How well do LLMs match syntactic properties of text domains?
di: Ju, Da, et al.
Pubblicazione: (2025)

KazMMLU: Evaluating Language Models on Kazakh, Russian, and Regional Knowledge of Kazakhstan
di: Togmanov, Mukhammed, et al.
Pubblicazione: (2025)

SinhalaMMLU: A Comprehensive Benchmark for Evaluating Multitask Language Understanding in Sinhala
di: Pramodya, Ashmari, et al.
Pubblicazione: (2025)

Mobile-MMLU: A Mobile Intelligence Language Understanding Benchmark
di: Bsharat, Sondos Mahmoud, et al.
Pubblicazione: (2025)

Model Unlearning Objectives Vary for Distinct Language Functions
di: Atil, Berk, et al.
Pubblicazione: (2026)

MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmark
di: Wang, Yubo, et al.
Pubblicazione: (2024)

GreekMMLU: A Native-Sourced Multitask Benchmark for Evaluating Language Models in Greek
di: Zhang, Yang, et al.
Pubblicazione: (2026)

MMLU-ProX: A Multilingual Benchmark for Advanced Large Language Model Evaluation
di: Xuan, Weihao, et al.
Pubblicazione: (2025)

Rethinking Dense Sequential Chains: Reasoning Language Models Can Extract Answers from Sparse, Order-Shuffling Chain-of-Thoughts
di: Chen, Yi-Chang, et al.
Pubblicazione: (2026)

What's in Common? Multimodal Models Hallucinate When Reasoning Across Scenes
di: Ross, Candace, et al.
Pubblicazione: (2025)

Khayyam Challenge (PersianMMLU): Is Your LLM Truly Wise to The Persian Language?
di: Ghahroodi, Omid, et al.
Pubblicazione: (2024)

Reactor Mk.1 performances: MMLU, HumanEval and BBH test results
di: Dunham, TJ, et al.
Pubblicazione: (2024)

DialectalArabicMMLU: Benchmarking Dialectal Capabilities in Arabic and Multilingual Language Models
di: Altakrori, Malik H., et al.
Pubblicazione: (2025)

BabyLM Turns 3: Call for papers for the 2025 BabyLM workshop
di: Charpentier, Lucas, et al.
Pubblicazione: (2025)

LLMs Can Generate a Better Answer by Aggregating Their Own Responses
di: Li, Zichong, et al.
Pubblicazione: (2025)

Reverse Question Answering: Can an LLM Write a Question so Hard (or Bad) that it Can't Answer?
di: Balepur, Nishant, et al.
Pubblicazione: (2024)

BioACE: An Automated Framework for Biomedical Answer and Citation Evaluations
di: Gupta, Deepak, et al.
Pubblicazione: (2026)

IndicMMLU-Pro: Benchmarking Indic Large Language Models on Multi-Task Language Understanding
di: KJ, Sankalp, et al.
Pubblicazione: (2025)

SWE-QA: Can Language Models Answer Repository-level Code Questions?
di: Peng, Weihan, et al.
Pubblicazione: (2025)

Semantic Consistency for Assuring Reliability of Large Language Models
di: Raj, Harsh, et al.
Pubblicazione: (2023)