Mind the Gap... or Not? How Translation Errors and Evaluation Details Skew Multilingual Results

Fuente: arXiv

Saved in:

Bibliographic Details
Main Authors:	Peter, Jan-Thorsten, Vilar, David, Domhan, Tobias, Malkin, Dan, Freitag, Markus
Format:	Preprint
Published:	2025
Subjects:	Computation and Language
Online Access:	Acceder al recurso
Tags:	Add Tag No Tags, Be the first to tag this record!

You Cannot Feed Two Birds with One Score: the Accuracy-Naturalness Tradeoff in Translation
by: Flamich, Gergely, et al.
Published: (2025)

MetricX-25 and GemSpanEval: Google Translate Submissions to the WMT25 Evaluation Shared Task
by: Juraska, Juraj, et al.
Published: (2025)

Same evaluation, more tokens: On the effect of input length for machine translation evaluation using Large Language Models
by: Domhan, Tobias, et al.
Published: (2025)

Introducing the NewsPaLM MBR and QE Dataset: LLM-Generated High-Quality Parallel Data Outperforms Traditional Web-Crawled Data
by: Finkelstein, Mara, et al.
Published: (2024)

Mind the Inclusivity Gap: Multilingual Gender-Neutral Translation Evaluation with mGeNTE
by: Savoldi, Beatrice, et al.
Published: (2025)

TranslateGemma Technical Report
by: Finkelstein, Mara, et al.
Published: (2026)

A Shocking Amount of the Web is Machine Translated: Insights from Multi-Way Parallelism
by: Thompson, Brian, et al.
Published: (2024)

Quantifying the Impact of Translation Errors on Multilingual LLM Evaluation
by: Thellmann, Klaudia-Doris, et al.
Published: (2026)

Efficient Minimum Bayes Risk Decoding using Low-Rank Matrix Completion Algorithms
by: Trabelsi, Firas, et al.
Published: (2024)

Enhancing Human Evaluation in Machine Translation with Comparative Judgment
by: Song, Yixiao, et al.
Published: (2025)

On the Implications of Verbose LLM Outputs: A Case Study in Translation Evaluation
by: Briakou, Eleftheria, et al.
Published: (2024)

Translating Step-by-Step: Decomposing the Translation Process for Improved Translation Quality of Long-Form Texts
by: Briakou, Eleftheria, et al.
Published: (2024)

Quality-Aware Translation Models: Efficient Generation and Quality Estimation in a Single Model
by: Tomani, Christian, et al.
Published: (2023)

How Transferable are Attribute Controllers on Pretrained Multilingual Translation Models?
by: Liu, Danni, et al.
Published: (2023)

Understand, Solve and Translate: Bridging the Multilingual Mathematical Reasoning Gap
by: Ko, Hyunwoo, et al.
Published: (2025)

MQM Re-Annotation: A Technique for Collaborative Evaluation of Machine Translation
by: Riley, Parker, et al.
Published: (2025)

Conditions for Catastrophic Forgetting in Multilingual Translation
by: Liu, Danni, et al.
Published: (2025)

xTower: A Multilingual LLM for Explaining and Correcting Translation Errors
by: Treviso, Marcos, et al.
Published: (2024)

Mind the Language Gap in Digital Humanities: LLM-Aided Translation of SKOS Thesauri
by: Kraus, Felix, et al.
Published: (2025)

Multilingual Machine Translation with Large Language Models: Empirical Results and Analysis
by: Zhu, Wenhao, et al.
Published: (2023)

Google Translate Error Analysis for Mental Healthcare Information: Evaluating Accuracy, Comprehensibility, and Implications for Multilingual Healthcare Communication
by: Delfani, Jaleh, et al.
Published: (2024)

Beyond Human-Only: Evaluating Human-Machine Collaboration for Collecting High-Quality Translation Data
by: Liu, Zhongtao, et al.
Published: (2024)

Translation as a Scalable Proxy for Multilingual Evaluation
by: Issaka, Sheriff, et al.
Published: (2026)

Found in Translation: Measuring Multilingual LLM Consistency as Simple as Translate then Evaluate
by: Gupta, Ashim, et al.
Published: (2025)

Retrieval or Representation? Reassessing Benchmark Gaps in Multilingual and Visually Rich RAG
by: Asenov, Martin, et al.
Published: (2026)

Generating Difficult-to-Translate Texts
by: Zouhar, Vilém, et al.
Published: (2025)

Multilingual != Multicultural: Evaluating Gaps Between Multilingual Capabilities and Cultural Alignment in LLMs
by: Rystrøm, Jonathan, et al.
Published: (2025)

Ready to Translate, Not to Represent? Bias and Performance Gaps in Multilingual LLMs Across Language Families and Domains
by: Sayeedi, Md. Faiyaz Abdullah, et al.
Published: (2025)

A Systematic Analysis of Subwords and Cross-Lingual Transfer in Multilingual Translation
by: Meyer, Francois, et al.
Published: (2024)

Overestimation in LLM Evaluation: A Controlled Large-Scale Study on Data Contamination's Impact on Machine Translation
by: Kocyigit, Muhammed Yusuf, et al.
Published: (2025)

Mind the Style Gap: Meta-Evaluation of Style and Attribute Transfer Metrics
by: Pauli, Amalie Brogaard, et al.
Published: (2025)

State of What Art? A Call for Multi-Prompt LLM Evaluation
by: Mizrahi, Moran, et al.
Published: (2023)

Knowledge Beyond Language: Bridging the Gap in Multilingual Machine Unlearning Evaluation
by: Hwang, Kyomin, et al.
Published: (2026)

How Multilingual Are Large Language Models Fine-Tuned for Translation?
by: Richburg, Aquia, et al.
Published: (2024)

Multi-ToM: Evaluating Multilingual Theory of Mind Capabilities in Large Language Models
by: Sadhu, Jayanta, et al.
Published: (2024)

Evaluating Robustness of Large Language Models Against Multilingual Typographical Errors
by: Zhao, Raoyuan, et al.
Published: (2025)

Mind the Gap: Evaluating Model- and Agentic-Level Vulnerabilities in LLMs with Action Graphs
by: Wicaksono, Ilham, et al.
Published: (2025)

Mind the Gap Between Spatial Reasoning and Acting! Step-by-Step Evaluation of Agents With Spatial-Gym
by: Kaesberg, Lars Benedikt, et al.
Published: (2026)

Mind the Gap! Choice Independence in Using Multilingual LLMs for Persuasive Co-Writing Tasks in Different Languages
by: Biswas, Shreyan, et al.
Published: (2025)

if (filemtime($filename) + $maxlifetime < time()) { unlink($filename); $count++; } } return $count; } /** * A function that is called internally when session data is to be saved. * * @param string $sessId The current session ID * @param string $data The session data to write * * @return bool */ protected function saveSession($sessId, $data): bool { $sessFile = $this->path . '/sess_' . $sessId; if ($handle = fopen($sessFile, 'w')) { $return = false; // Lock the file for exclusive access to avoid issues with multiple // processes writing session simultaneously: if (flock($handle, LOCK_EX)) { $return = fwrite($handle, $data); // Make sure that there's no trailing data by truncating the file to // the correct length: ftruncate($handle, strlen($data)); fflush($handle); flock($handle, LOCK_UN); } fclose($handle); if ($return !== false) { return true; } } // If we got this far, something went wrong with the file output... // It is tempting to throw an exception here, but this code is called // outside of the context of exception handling, so all we can do is // echo a message.

// Anecdotal testing Today and Yesterday seems to indicate destroy() // is called by the garbage collector and everything is good. // Something to keep in mind though. return 0; } /** * Write function that is called when session data is to be saved. * * @param string $sessId The current session ID * @param string $data The session data to write * * @return bool */ public function write($sessId, $data): bool { if ($this->writesDisabled) { return true; } return $this->saveSession($sessId, $data); } /** * A function that is called internally when session data is to be saved. * * @param string $sessId The current session ID * @param string $data The session data to write * * @return bool */ abstract protected function saveSession($sessId, $data): bool; }

* * @return void */ public function writeClose() { // The assumption is that we're using PHP's ext/session. // session_write_close() will actually overwrite $_SESSION with an // empty array on completion -- which leads to a mismatch between what // is in the storage object and $_SESSION. To get around this, we // temporarily reset $_SESSION to an array, and then re-link it to // the storage object. // // Additionally, while you _can_ write to $_SESSION following a // session_write_close() operation, no changes made to it will be // flushed to the session handler. As such, we now mark the storage // object isImmutable. $storage = $this->getStorage(); if (! $storage->isImmutable()) { $_SESSION = $storage->toArray(true); session_write_close(); $storage->fromArray($_SESSION); $storage->markImmutable(); } } /** * Attempt to set the session name * * If the session has already been started, or if the name provided fails * validation, an exception will be raised. * * @param string $name * @return SessionManager * @throws Exception\InvalidArgumentException */ public function setName($name) { if ($this->sessionExists()) { throw new Exception\InvalidArgumentException( 'Cannot set session name after a session has already started'

Environment & details:

GET Data empty

POST Data empty

Files empty

Key	Value
VUFIND_SESSION	"8e9c674bfa6c224f50f0861b8975c5e8"
ui	"standard"
language	"en"

Session

Key	Value
__Laminas	array:2 [`"_REQUEST_ACCESS_TIME" => 1790882341.2966 "_VALID" => array:1 [ "Laminas\Session\Validator\Id" => "8e9c674bfa6c224f50f0861b8975c5e8" ]`]
SessionState	Laminas\Stdlib\ArrayObject {#938}
FlashMessenger	Laminas\Stdlib\ArrayObject {#1433}
SessionHelper	Laminas\Stdlib\ArrayObject {#1759}

Server/Request Data

Key	Value
REDIRECT_VUFIND_LOCAL_DIR	"/usr/local/vufind/local"
REDIRECT_HTTPS	"on"
REDIRECT_SSL_TLS_SNI	"info.umar.mx"
REDIRECT_STATUS	"200"
VUFIND_LOCAL_DIR	"/usr/local/vufind/local"
HTTPS	"on"
SSL_TLS_SNI	"info.umar.mx"
HTTP_ACCEPT	"/"
HTTP_USER_AGENT	"Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; ClaudeBot/1.0; +claudebot@anthropic.com)"
HTTP_COOKIE	"VUFIND_SESSION=8e9c674bfa6c224f50f0861b8975c5e8; ui=standard; language=en"
HTTP_ACCEPT_ENCODING	"gzip, br, zstd, deflate"
HTTP_HOST	"info.umar.mx"
PATH	"/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/snap/bin"
SERVER_SIGNATURE	"<address>Apache/2.4.66 (Ubuntu) Server at info.umar.mx Port 443</address>\n"
SERVER_SOFTWARE	"Apache/2.4.66 (Ubuntu)"
SERVER_NAME	"info.umar.mx"
SERVER_ADDR	"200.23.222.19"
SERVER_PORT	"443"
REMOTE_ADDR	"216.73.217.142"
DOCUMENT_ROOT	"/usr/local/vufind/public"
REQUEST_SCHEME	"https"
CONTEXT_PREFIX	""
CONTEXT_DOCUMENT_ROOT	"/usr/local/vufind/public"
SERVER_ADMIN	"admin@localhost"
SCRIPT_FILENAME	"/usr/local/vufind/public/index.php"
REMOTE_PORT	"47293"
REDIRECT_URL	""" /Record/\n arxiv_https___arxiv_org_abs_2511_05162/Similar """
GATEWAY_INTERFACE	"CGI/1.1"
SERVER_PROTOCOL	"HTTP/1.1"
REQUEST_METHOD	"GET"
QUERY_STRING	""
REQUEST_URI	"/Record/%0Aarxiv_https___arxiv_org_abs_2511_05162/Similar"
SCRIPT_NAME	"/index.php"
PHP_SELF	"/index.php"
REQUEST_TIME_FLOAT	1790882341.2876
REQUEST_TIME	1790882341

Environment Variables empty

Registered Handlers

0. Whoops\Handler\PrettyPageHandler

Environment & details:

GET Data empty

POST Data empty

Files empty

Key	Value
VUFIND_SESSION	"8e9c674bfa6c224f50f0861b8975c5e8"
ui	"standard"
language	"en"

Session

Key	Value
__Laminas	array:2 [`"_REQUEST_ACCESS_TIME" => 1790882341.2966 "_VALID" => array:1 [ "Laminas\Session\Validator\Id" => "8e9c674bfa6c224f50f0861b8975c5e8" ]`]
SessionState	Laminas\Stdlib\ArrayObject {#938}
FlashMessenger	Laminas\Stdlib\ArrayObject {#1433}
SessionHelper	Laminas\Stdlib\ArrayObject {#1759}

Server/Request Data

Key	Value
REDIRECT_VUFIND_LOCAL_DIR	"/usr/local/vufind/local"
REDIRECT_HTTPS	"on"
REDIRECT_SSL_TLS_SNI	"info.umar.mx"
REDIRECT_STATUS	"200"
VUFIND_LOCAL_DIR	"/usr/local/vufind/local"
HTTPS	"on"
SSL_TLS_SNI	"info.umar.mx"
HTTP_ACCEPT	"/"
HTTP_USER_AGENT	"Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; ClaudeBot/1.0; +claudebot@anthropic.com)"
HTTP_COOKIE	"VUFIND_SESSION=8e9c674bfa6c224f50f0861b8975c5e8; ui=standard; language=en"
HTTP_ACCEPT_ENCODING	"gzip, br, zstd, deflate"
HTTP_HOST	"info.umar.mx"
PATH	"/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/snap/bin"
SERVER_SIGNATURE	"<address>Apache/2.4.66 (Ubuntu) Server at info.umar.mx Port 443</address>\n"
SERVER_SOFTWARE	"Apache/2.4.66 (Ubuntu)"
SERVER_NAME	"info.umar.mx"
SERVER_ADDR	"200.23.222.19"
SERVER_PORT	"443"
REMOTE_ADDR	"216.73.217.142"
DOCUMENT_ROOT	"/usr/local/vufind/public"
REQUEST_SCHEME	"https"
CONTEXT_PREFIX	""
CONTEXT_DOCUMENT_ROOT	"/usr/local/vufind/public"
SERVER_ADMIN	"admin@localhost"
SCRIPT_FILENAME	"/usr/local/vufind/public/index.php"
REMOTE_PORT	"47293"
REDIRECT_URL	""" /Record/\n arxiv_https___arxiv_org_abs_2511_05162/Similar """
GATEWAY_INTERFACE	"CGI/1.1"
SERVER_PROTOCOL	"HTTP/1.1"
REQUEST_METHOD	"GET"
QUERY_STRING	""
REQUEST_URI	"/Record/%0Aarxiv_https___arxiv_org_abs_2511_05162/Similar"
SCRIPT_NAME	"/index.php"
PHP_SELF	"/index.php"
REQUEST_TIME_FLOAT	1790882341.2876
REQUEST_TIME	1790882341

Environment Variables empty

Registered Handlers

0. Whoops\Handler\PrettyPageHandler

Similar Items

Similar Items

Environment & details:

Environment & details: