Cover Image

Pretrained Multilingual Transformers Reveal Quantitative Distance Between Human Languages

Fuente: arXiv

Saved in:

Bibliographic Details
Main Authors:	Zhao, Yue, Gu, Jiatao, Jeretič, Paloma, Su, Weijie
Format:	Preprint
Published:	2026
Subjects:	Computation and Language Machine Learning
Online Access:	Acceder al recurso
Tags:	Add Tag No Tags, Be the first to tag this record!

Similar Items

Factual Consistency of Multilingual Pretrained Language Models
by: Fierro, Constanza, et al.
Published: (2022)

Revisiting Multilingual Data Mixtures in Language Model Pretraining
by: Foroutan, Negar, et al.
Published: (2025)

Identifying the Correlation Between Language Distance and Cross-Lingual Transfer in a Multilingual Representation Space
by: Philippy, Fred, et al.
Published: (2023)

Flexible Language Modeling in Continuous Space with Transformer-based Autoregressive Flows
by: Zhang, Ruixiang, et al.
Published: (2025)

ATLAS: Adaptive Transfer Scaling Laws for Multilingual Pretraining, Finetuning, and Decoding the Curse of Multilinguality
by: Longpre, Shayne, et al.
Published: (2025)

Enhancing Multilingual LLM Pretraining with Model-Based Data Selection
by: Messmer, Bettina, et al.
Published: (2025)

Length-MAX Tokenizer for Language Models
by: Dong, Dong, et al.
Published: (2025)

LDP: Generalizing to Multilingual Visual Information Extraction by Language Decoupled Pretraining
by: Shen, Huawen, et al.
Published: (2024)

Judging Quality Across Languages: A Multilingual Approach to Pretraining Data Filtering with Language Models
by: Ali, Mehdi, et al.
Published: (2025)

MuRating: A High Quality Data Selecting Approach to Multilingual Large Language Model Pretraining
by: Chen, Zhixun, et al.
Published: (2025)

Learning When to Think While Listening in Large Audio-Language Models
by: Song, Zhiyuan, et al.
Published: (2026)

A Law of Next-Token Prediction in Large Language Models
by: He, Hangfeng, et al.
Published: (2024)

Scaling Laws for Multilingual Language Models
by: He, Yifei, et al.
Published: (2024)

TPTT: Transforming Pretrained Transformers into Titans
by: Furfaro, Fabien
Published: (2025)

Robust Detection of Watermarks for Large Language Models Under Human Edits
by: Li, Xiang, et al.
Published: (2024)

APT: Adaptive Pruning and Tuning Pretrained Language Models for Efficient Training and Inference
by: Zhao, Bowen, et al.
Published: (2024)

Born a Transformer -- Always a Transformer? On the Effect of Pretraining on Architectural Abilities
by: Jobanputra, Mayank, et al.
Published: (2025)

Discovering Low-rank Subspaces for Language-agnostic Multilingual Representations
by: Xie, Zhihui, et al.
Published: (2024)

Disentangling Language Roles in Multilingual LLM Task Execution
by: Zhan, Qishi, et al.
Published: (2026)

Comparison of Scoring Rationales Between Large Language Models and Human Raters
by: Hua, Haowei, et al.
Published: (2025)

Look Ahead or Look Around? A Theoretical Comparison Between Autoregressive and Masked Pretraining
by: Zhang, Qi, et al.
Published: (2024)

TyphoFormer: Language-Augmented Transformer for Accurate Typhoon Track Forecasting
by: Li, Lincan, et al.
Published: (2025)

Optimal Estimation of Watermark Proportions in Hybrid AI-Human Texts
by: Li, Xiang, et al.
Published: (2025)

Unsupervised Pretraining for Fact Verification by Language Model Distillation
by: Bazaga, Adrián, et al.
Published: (2023)

Multilingual Needle in a Haystack: Investigating Long-Context Behavior of Multilingual Large Language Models
by: Hengle, Amey, et al.
Published: (2024)

Divide-or-Conquer? Which Part Should You Distill Your LLM?
by: Wu, Zhuofeng, et al.
Published: (2024)

Procedural Pretraining: Warming Up Language Models with Abstract Data
by: Jiang, Liangze, et al.
Published: (2026)

Facts in Stats: Impacts of Pretraining Diversity on Language Model Generalization
by: Behnia, Tina, et al.
Published: (2025)

The Heuristic Core: Understanding Subnetwork Generalization in Pretrained Language Models
by: Bhaskar, Adithya, et al.
Published: (2024)

NoteContrast: Contrastive Language-Diagnostic Pretraining for Medical Text
by: Kailas, Prajwal, et al.
Published: (2024)

ESLM: Risk-Averse Selective Language Modeling for Efficient Pretraining
by: Bal, Melis Ilayda, et al.
Published: (2025)

Procedural Knowledge in Pretraining Drives Reasoning in Large Language Models
by: Ruis, Laura, et al.
Published: (2024)

Analyzing Similarity Metrics for Data Selection for Language Model Pretraining
by: Sam, Dylan, et al.
Published: (2025)

Repeated-Token Counting Reveals a Dissociation Between Representations and Outputs
by: Venkatesh, Sohan
Published: (2026)

Can Brain Signals Reveal Inner Alignment with Human Languages?
by: Han, William, et al.
Published: (2022)

Data Mixing for Large Language Models Pretraining: A Survey and Outlook
by: Chen, Zhuo, et al.
Published: (2026)

RePro: Training Language Models to Faithfully Recycle the Web for Pretraining
by: Yu, Zichun, et al.
Published: (2025)

Language Models Improve When Pretraining Data Matches Target Tasks
by: Mizrahi, David, et al.
Published: (2025)

Task-Adaptive Pretrained Language Models via Clustered-Importance Sampling
by: Grangier, David, et al.
Published: (2024)

Temporal Entailment Pretraining for Clinical Language Models over EHR Data
by: Tanaka, Tatsunori, et al.
Published: (2025)