Cover Image

MEDAL: A Framework for Benchmarking LLMs as Multilingual Open-Domain Dialogue Evaluators

Fuente: arXiv

Saved in:

Bibliographic Details
Main Authors:	Mendonça, John, Lavie, Alon, Trancoso, Isabel
Format:	Preprint
Published:	2025
Subjects:	Computation and Language
Online Access:	Acceder al recurso
Tags:	Add Tag No Tags, Be the first to tag this record!

Similar Items

On the Benchmarking of LLMs for Open-Domain Dialogue Evaluation
by: Mendonça, John, et al.
Published: (2024)

Soda-Eval: Open-Domain Dialogue Evaluation in the age of LLMs
by: Mendonça, John, et al.
Published: (2024)

ECoh: Turn-level Coherence Evaluation for Multilingual Dialogues
by: Mendonça, John, et al.
Published: (2024)

Overview of Dialog System Evaluation Track: Dimensionality, Language, Culture and Safety at DSTC 12
by: Mendonça, John, et al.
Published: (2025)

Simulated Reasoning is Reasoning
by: Kempt, Hendrik, et al.
Published: (2026)

SLIDE: A Framework Integrating Small and Large Language Models for Open-Domain Dialogues Evaluation
by: Zhao, Kun, et al.
Published: (2024)

Emphasising Structured Information: Integrating Abstract Meaning Representation into LLMs for Enhanced Open-Domain Dialogue Evaluation
by: Yang, Bohao, et al.
Published: (2024)

Modeling the One-to-Many Property in Open-Domain Dialogue with LLMs
by: Lee, Jing Yang, et al.
Published: (2025)

XLQA: A Benchmark for Locale-Aware Multilingual Open-Domain Question Answering
by: Roh, Keon-Woo, et al.
Published: (2025)

SumRec: A Framework for Recommendation using Open-Domain Dialogue
by: Asahara, Ryutaro, et al.
Published: (2024)

Rethinking Response Evaluation from Interlocutor's Eye for Open-Domain Dialogue Systems
by: Tsuta, Yuma, et al.
Published: (2024)

OMGEval: An Open Multilingual Generative Evaluation Benchmark for Large Language Models
by: Liu, Yang, et al.
Published: (2024)

MMAFFBen: A Multilingual and Multimodal Affective Analysis Benchmark for Evaluating LLMs and VLMs
by: Liu, Zhiwei, et al.
Published: (2025)

P-MMEval: A Parallel Multilingual Multitask Benchmark for Consistent Evaluation of LLMs
by: Zhang, Yidan, et al.
Published: (2024)

MCEval: A Dynamic Framework for Fair Multilingual Cultural Evaluation of LLMs
by: Huang, Shulin, et al.
Published: (2025)

CricBench: A Multilingual Benchmark for Evaluating LLMs in Cricket Analytics
by: Agarwal, Parth, et al.
Published: (2025)

StereoTales: A Multilingual Framework for Open-Ended Stereotype Discovery in LLMs
by: Jeune, Pierre Le, et al.
Published: (2026)

Multilingual Dialogue Generation and Localization with Dialogue Act Scripting
by: Vasselli, Justin, et al.
Published: (2025)

MultiNRC: A Challenging and Native Multilingual Reasoning Evaluation Benchmark for LLMs
by: Fabbri, Alexander R., et al.
Published: (2025)

IndicGenBench: A Multilingual Benchmark to Evaluate Generation Capabilities of LLMs on Indic Languages
by: Singh, Harman, et al.
Published: (2024)

Evaluating LLMs' Multilingual Capabilities for Bengali: Benchmark Creation and Performance Analysis
by: Bhowmik, Shimanto, et al.
Published: (2025)

OMIND: Framework for Knowledge Grounded Finetuning and Multi-Turn Dialogue Benchmark for Mental Health LLMs
by: Racha, Suraj, et al.
Published: (2026)

CAMÕES: A Comprehensive Automatic Speech Recognition Benchmark for European Portuguese
by: Carvalho, Carlos, et al.
Published: (2025)

Open-Source Large Language Models as Multilingual Crowdworkers: Synthesizing Open-Domain Dialogues in Several Languages With No Examples in Targets and No Machine Translation
by: Njifenjou, Ahmed, et al.
Published: (2025)

DRE: An Effective Dual-Refined Method for Integrating Small and Large Language Models in Open-Domain Dialogue Evaluation
by: Zhao, Kun, et al.
Published: (2025)

MAKIEval: A Multilingual Automatic WiKidata-based Framework for Cultural Awareness Evaluation for LLMs
by: Zhao, Raoyuan, et al.
Published: (2025)

Benchmarking Multilingual Speech Models on Pashto: Zero-Shot ASR, Script Failure, and Cross-Domain Evaluation
by: Rahman, Hanif
Published: (2026)

A Novel Evaluation Benchmark for Medical LLMs: Illuminating Safety and Effectiveness in Clinical Domains
by: Wang, Shirui, et al.
Published: (2025)

Multi-IF: Benchmarking LLMs on Multi-Turn and Multilingual Instructions Following
by: He, Yun, et al.
Published: (2024)

I Am Aligned, But With Whom? MENA Values Benchmark for Evaluating Cultural Alignment and Multilingual Bias in LLMs
by: Zahraei, Pardis Sadat, et al.
Published: (2025)

Dynamic Stochastic Decoding Strategy for Open-Domain Dialogue Generation
by: Li, Yiwei, et al.
Published: (2024)

A Framework to Assess Multilingual Vulnerabilities of LLMs
by: Tang, Likai, et al.
Published: (2025)

The GaoYao Benchmark: A Comprehensive Framework for Evaluating Multilingual and Multicultural Abilities of Large Language Models
by: Liu, Yilun, et al.
Published: (2026)

MFTCXplain: A Multilingual Benchmark Dataset for Evaluating the Moral Reasoning of LLMs through Multi-hop Hate Speech Explanation
by: Trager, Jackson, et al.
Published: (2025)

Challenging Multilingual LLMs: A New Taxonomy and Benchmark for Unraveling Hallucination in Translation
by: Wu, Xinwei, et al.
Published: (2025)

PairEval: Open-domain Dialogue Evaluation with Pairwise Comparison
by: Park, ChaeHun, et al.
Published: (2024)

ATOD: An Evaluation Framework and Benchmark for Agentic Task-Oriented Dialogue Systems
by: Zhang, Yifei, et al.
Published: (2026)

Cross-Lingual Auto Evaluation for Assessing Multilingual LLMs
by: Doddapaneni, Sumanth, et al.
Published: (2024)

The AI Language Proficiency Monitor -- Tracking the Progress of LLMs on Multilingual Benchmarks
by: Pomerenke, David, et al.
Published: (2025)

"Be My Cheese?": Cultural Nuance Benchmarking for Machine Translation in Multilingual LLMs
by: Van Doren, Madison, et al.
Published: (2026)