Image de couverture de livre

Finding Replicable Human Evaluations via Stable Ranking Probability

Fuente: arXiv

Enregistré dans:

Détails bibliographiques
Auteurs principaux:	Riley, Parker, Deutsch, Daniel, Foster, George, Ratnakar, Viresh, Dabirmoghaddam, Ali, Freitag, Markus
Format:	Preprint
Publié:	2024
Sujets:	Computation and Language
Accès en ligne:	Acceder al recurso
Tags:	Ajouter un tag Pas de tags, Soyez le premier à ajouter un tag!

Documents similaires

Enhancing Human Evaluation in Machine Translation with Comparative Judgment
par: Song, Yixiao, et autres
Publié: (2025)

Beyond Human-Only: Evaluating Human-Machine Collaboration for Collecting High-Quality Translation Data
par: Liu, Zhongtao, et autres
Publié: (2024)

MQM Re-Annotation: A Technique for Collaborative Evaluation of Machine Translation
par: Riley, Parker, et autres
Publié: (2025)

From Jack of All Trades to Master of One: Specializing LLM-based Autoraters to a Test Set
par: Finkelstein, Mara, et autres
Publié: (2024)

Generating Difficult-to-Translate Texts
par: Zouhar, Vilém, et autres
Publié: (2025)

Mitigating Metric Bias in Minimum Bayes Risk Decoding
par: Kovacs, Geza, et autres
Publié: (2024)

MetricX-24: The Google Submission to the WMT 2024 Metrics Shared Task
par: Juraska, Juraj, et autres
Publié: (2024)

When LLMs Benchmark Themselves: Deconstructing Self-Bias in Automated Evaluation
par: Xu, Wenda, et autres
Publié: (2025)

Overestimation in LLM Evaluation: A Controlled Large-Scale Study on Data Contamination's Impact on Machine Translation
par: Kocyigit, Muhammed Yusuf, et autres
Publié: (2025)

MetricX-25 and GemSpanEval: Google Translate Submissions to the WMT25 Evaluation Shared Task
par: Juraska, Juraj, et autres
Publié: (2025)

Improving Statistical Significance in Human Evaluation of Automatic Metrics via Soft Pairwise Accuracy
par: Thompson, Brian, et autres
Publié: (2024)

LLMRefine: Pinpointing and Refining Large Language Models via Fine-Grained Actionable Feedback
par: Xu, Wenda, et autres
Publié: (2023)

The Chameleon Nature of LLMs: Quantifying Multi-Turn Stance Instability in Search-Enabled Language Models
par: Ratnakar, Shivam, et autres
Publié: (2025)

Don't Sweat the Small Stuff: Segment-Level Meta-Evaluation Based on Pairwise Difference Correlation
par: DiIanni, Colten, et autres
Publié: (2025)

WMT24++: Expanding the Language Coverage of WMT24 to 55 Languages & Dialects
par: Deutsch, Daniel, et autres
Publié: (2025)

Efficient Minimum Bayes Risk Decoding using Low-Rank Matrix Completion Algorithms
par: Trabelsi, Firas, et autres
Publié: (2024)

On the Implications of Verbose LLM Outputs: A Case Study in Translation Evaluation
par: Briakou, Eleftheria, et autres
Publié: (2024)

To Diverge or Not to Diverge: A Morphosyntactic Perspective on Machine Translation vs Human Translation
par: Luo, Jiaming, et autres
Publié: (2024)

STARLING: Self-supervised Training of Text-based Reinforcement Learning Agent with Large Language Models
par: Basavatia, Shreyas, et autres
Publié: (2024)

Introducing the NewsPaLM MBR and QE Dataset: LLM-Generated High-Quality Parallel Data Outperforms Traditional Web-Crawled Data
par: Finkelstein, Mara, et autres
Publié: (2024)

Mind the Gap... or Not? How Translation Errors and Evaluation Details Skew Multilingual Results
par: Peter, Jan-Thorsten, et autres
Publié: (2025)

TranslateGemma Technical Report
par: Finkelstein, Mara, et autres
Publié: (2026)

Preliminary Ranking of WMT25 General Machine Translation Systems
par: Kocmi, Tom, et autres
Publié: (2025)

TyDi QA-WANA: A Benchmark for Information-Seeking Question Answering in Languages of West Asia and North Africa
par: Riley, Parker, et autres
Publié: (2025)

Replicating Human Social Perception in Generative AI: Evaluating the Valence-Dominance Model
par: Gurkan, Necdet, et autres
Publié: (2025)

Do LLMs Find Human Answers To Fact-Driven Questions Perplexing? A Case Study on Reddit
par: Seegmiller, Parker, et autres
Publié: (2024)

HumanRankEval: Automatic Evaluation of LMs as Conversational Assistants
par: Gritta, Milan, et autres
Publié: (2024)

Translating Step-by-Step: Decomposing the Translation Process for Improved Translation Quality of Long-Form Texts
par: Briakou, Eleftheria, et autres
Publié: (2024)

Left, Right, or Center? Evaluating LLM Framing in News Classification and Generation
par: Kennedy, Molly, et autres
Publié: (2026)

Beyond QA Pairs: Assessing Parameter-Efficient Fine-Tuning for Fact Embedding in LLMs
par: Ratnakar, Shivam, et autres
Publié: (2025)

You Cannot Feed Two Birds with One Score: the Accuracy-Naturalness Tradeoff in Translation
par: Flamich, Gergely, et autres
Publié: (2025)

Language Models can Evaluate Themselves via Probability Discrepancy
par: Xia, Tingyu, et autres
Publié: (2024)

Penalizing Length: Uncovering Systematic Bias in Quality Estimation Metrics
par: Zhang, Yilin, et autres
Publié: (2025)

MBR and QE Finetuning: Training-time Distillation of the Best and Most Expensive Decoding Methods
par: Finkelstein, Mara, et autres
Publié: (2023)

LLMs as Workers in Human-Computational Algorithms? Replicating Crowdsourcing Pipelines with LLMs
par: Wu, Tongshuang, et autres
Publié: (2023)

Feeding Two Birds or Favoring One? Adequacy-Fluency Tradeoffs in Evaluation and Meta-Evaluation of Machine Translation
par: Shayegh, Behzad, et autres
Publié: (2025)

Interpreting Predictive Probabilities: Model Confidence or Human Label Variation?
par: Baan, Joris, et autres
Publié: (2024)

Findings of the IWSLT 2024 Evaluation Campaign
par: Ahmad, Ibrahim Said, et autres
Publié: (2024)

PaperBench: Evaluating AI's Ability to Replicate AI Research
par: Starace, Giulio, et autres
Publié: (2025)

Preliminary WMT24 Ranking of General MT Systems and LLMs
par: Kocmi, Tom, et autres
Publié: (2024)