Image de couverture de livre

Explaining and Mitigating Crosslingual Tokenizer Inequities

Fuente: arXiv

Enregistré dans:

Détails bibliographiques
Auteurs principaux:	Arnett, Catherine, Chang, Tyler A., Biderman, Stella, Bergen, Benjamin K.
Format:	Preprint
Publié:	2025
Sujets:	Computation and Language
Accès en ligne:	Acceder al recurso
Tags:	Ajouter un tag Pas de tags, Soyez le premier à ajouter un tag!

Documents similaires

A Bit of a Problem: Measurement Disparities in Dataset Sizes Across Languages
par: Arnett, Catherine, et autres
Publié: (2024)

Goldfish: Monolingual Language Models for 350 Languages
par: Chang, Tyler A., et autres
Publié: (2024)

Why do language models perform worse for morphologically complex languages?
par: Arnett, Catherine, et autres
Publié: (2024)

On the Acquisition of Shared Grammatical Representations in Bilingual Language Models
par: Arnett, Catherine, et autres
Publié: (2025)

Bigram Subnetworks: Mapping to Next Tokens in Transformer Language Models
par: Chang, Tyler A., et autres
Publié: (2025)

Revenge of the Fallen? Recurrent Models Match Transformers at Predicting Human Language Comprehension Metrics
par: Michaelov, James A., et autres
Publié: (2024)

Different Tokenization Schemes Lead to Comparable Performance in Spanish Number Agreement
par: Arnett, Catherine, et autres
Publié: (2024)

Characterizing Learning Curves During Language Model Pre-Training: Learning, Forgetting, and Stability
par: Chang, Tyler A., et autres
Publié: (2023)

Evaluating Morphological Alignment of Tokenizers in 70 Languages
par: Arnett, Catherine, et autres
Publié: (2025)

Weight Tying Biases Token Embeddings Towards the Output Space
par: Lopardo, Antonio, et autres
Publié: (2026)

BPE Gets Picky: Efficient Vocabulary Refinement During Tokenizer Training
par: Chizhov, Pavel, et autres
Publié: (2024)

Disaggregation Reveals Hidden Training Dynamics: The Case of Agreement Attraction
par: Michaelov, James A., et autres
Publié: (2025)

BPE Stays on SCRIPT: Structured Encoding for Robust Multilingual Pretokenization
par: Land, Sander, et autres
Publié: (2025)

How Open Must Language Models be to Enable Reliable Scientific Inference?
par: Michaelov, James A., et autres
Publié: (2026)

Crosslingual On-Policy Self-Distillation for Multilingual Reasoning
par: Liu, Yihong, et autres
Publié: (2026)

Mitigating the Risk of Health Inequity Exacerbated by Large Language Models
par: Ji, Yuelyu, et autres
Publié: (2024)

On the Entity-Level Alignment in Crosslingual Consistency
par: Liu, Yihong, et autres
Publié: (2025)

How Transliterations Improve Crosslingual Alignment
par: Liu, Yihong, et autres
Publié: (2024)

Emergent inabilities? Inverse scaling over the course of pretraining
par: Michaelov, James A., et autres
Publié: (2023)

LLM Circuit Analyses Are Consistent Across Training and Scale
par: Tigges, Curt, et autres
Publié: (2024)

Do Large Language Models Exhibit Spontaneous Rational Deception?
par: Taylor, Samuel M., et autres
Publié: (2025)

Crosslingual Optimized Metric for Translation Assessment of Indian Languages
par: Ahsan, Arafat, et autres
Publié: (2025)

An Interpretable and Crosslingual Method for Evaluating Second-Language Dialogues
par: Gao, Rena, et autres
Publié: (2024)

Large Language Models Pass the Turing Test
par: Jones, Cameron R., et autres
Publié: (2025)

Distilling Monolingual and Crosslingual Word-in-Context Representations
par: Arase, Yuki, et autres
Publié: (2024)

Post-Training Language Models for Crosslingual Consistency
par: Liu, Tianyu, et autres
Publié: (2026)

EVOKE: Emotion Vocabulary Of Korean and English
par: Jung, Yoonwon, et autres
Publié: (2026)

Does GPT-4 pass the Turing test?
par: Jones, Cameron R., et autres
Publié: (2023)

Transformer-Based Models Are Not Yet Perfect At Learning to Emulate Structural Recursion
par: Zhang, Dylan, et autres
Publié: (2024)

Language Model Behavioral Phases are Consistent Across Architecture, Training Data, and Scale
par: Michaelov, James A., et autres
Publié: (2025)

Suppressing Pink Elephants with Direct Principle Feedback
par: Castricato, Louis, et autres
Publié: (2024)

Crosslingual Reasoning through Test-Time Scaling
par: Yong, Zheng-Xin, et autres
Publié: (2025)

Crosslingual Capabilities and Knowledge Barriers in Multilingual Large Language Models
par: Chua, Lynn, et autres
Publié: (2024)

Multilingual vs Crosslingual Retrieval of Fact-Checked Claims: A Tale of Two Approaches
par: Ramponi, Alan, et autres
Publié: (2025)

Entropy2Vec: Crosslingual Language Modeling Entropy as End-to-End Learnable Language Representations
par: Irawan, Patrick Amadeus, et autres
Publié: (2025)

UWBa at SemEval-2025 Task 7: Multilingual and Crosslingual Fact-Checked Claim Retrieval
par: Lenc, Ladislav, et autres
Publié: (2025)

Dissecting the Ullman Variations with a SCALPEL: Why do LLMs fail at Trivial Alterations to the False Belief Task?
par: Pi, Zhiqiang, et autres
Publié: (2024)

Toxicity of the Commons: Curating Open-Source Pre-Training Data
par: Arnett, Catherine, et autres
Publié: (2024)

LEACE: Perfect linear concept erasure in closed form
par: Belrose, Nora, et autres
Publié: (2023)

Not quite Sherlock Holmes: Language model predictions do not reliably differentiate impossible from improbable events
par: Michaelov, James A., et autres
Publié: (2025)