Image de couverture de livre

Different Tokenization Schemes Lead to Comparable Performance in Spanish Number Agreement

Fuente: arXiv

Enregistré dans:

Détails bibliographiques
Auteurs principaux:	Arnett, Catherine, Rivière, Pamela D., Chang, Tyler A., Trott, Sean
Format:	Preprint
Publié:	2024
Sujets:	Computation and Language
Accès en ligne:	Acceder al recurso
Tags:	Ajouter un tag Pas de tags, Soyez le premier à ajouter un tag!

Documents similaires

Measuring and Modifying the Readability of English Texts with GPT-4
par: Trott, Sean, et autres
Publié: (2024)

Capacity Constraints and the Multilingual Penalty for Lexical Disambiguation
par: Trott, Sean, et autres
Publié: (2026)

Start Making Sense(s): A Developmental Probe of Attention Specialization Using Lexical Ambiguity
par: Rivière, Pamela D., et autres
Publié: (2025)

Evaluating Contextualized Representations of (Spanish) Ambiguous Words: A New Lexical Resource and Empirical Analysis
par: Rivière, Pamela D., et autres
Publié: (2024)

Explaining and Mitigating Crosslingual Tokenizer Inequities
par: Arnett, Catherine, et autres
Publié: (2025)

Disaggregation Reveals Hidden Training Dynamics: The Case of Agreement Attraction
par: Michaelov, James A., et autres
Publié: (2025)

How Open Must Language Models be to Enable Reliable Scientific Inference?
par: Michaelov, James A., et autres
Publié: (2026)

Language Statistics and False Belief Reasoning: Evidence from 41 Open-Weight LMs
par: Trott, Sean, et autres
Publié: (2026)

A Bit of a Problem: Measurement Disparities in Dataset Sizes Across Languages
par: Arnett, Catherine, et autres
Publié: (2024)

Toward a Theory of Generalizability in LLM Mechanistic Interpretability Research
par: Trott, Sean
Publié: (2025)

Evaluating Morphological Alignment of Tokenizers in 70 Languages
par: Arnett, Catherine, et autres
Publié: (2025)

Goldfish: Monolingual Language Models for 350 Languages
par: Chang, Tyler A., et autres
Publié: (2024)

On the Acquisition of Shared Grammatical Representations in Bilingual Language Models
par: Arnett, Catherine, et autres
Publié: (2025)

Weight Tying Biases Token Embeddings Towards the Output Space
par: Lopardo, Antonio, et autres
Publié: (2026)

BPE Gets Picky: Efficient Vocabulary Refinement During Tokenizer Training
par: Chizhov, Pavel, et autres
Publié: (2024)

Do language models capture implied discourse meanings? An investigation with exhaustivity implicatures of Korean morphology
par: Shin, Hagyeong, et autres
Publié: (2024)

BPE Stays on SCRIPT: Structured Encoding for Robust Multilingual Pretokenization
par: Land, Sander, et autres
Publié: (2025)

Why do language models perform worse for morphologically complex languages?
par: Arnett, Catherine, et autres
Publié: (2024)

Bigram Subnetworks: Mapping to Next Tokens in Transformer Language Models
par: Chang, Tyler A., et autres
Publié: (2025)

Revenge of the Fallen? Recurrent Models Match Transformers at Predicting Human Language Comprehension Metrics
par: Michaelov, James A., et autres
Publié: (2024)

Seeing Through Words, Speaking Through Pixels: Deep Representational Alignment Between Vision and Language Models
par: He, Zoe Wanying, et autres
Publié: (2025)

Toxicity of the Commons: Curating Open-Source Pre-Training Data
par: Arnett, Catherine, et autres
Publié: (2024)

Token-level Ensembling of Models with Different Vocabularies
par: Wicks, Rachel, et autres
Publié: (2025)

AI-Augmented Predictions: LLM Assistants Improve Human Forecasting Accuracy
par: Schoenegger, Philipp, et autres
Publié: (2024)

Re-defining Humor Data Objects for AI Humor Research
par: Arnett, Anna, et autres
Publié: (2026)

Regional Tiny Stories: Using Small Models to Compare Language Learning and Tokenizer Performance
par: Patil, Nirvan, et autres
Publié: (2025)

A Triadic Suffix Tokenization Scheme for Numerical Reasoning
par: Chetverina, Olga
Publié: (2026)

Broken Words, Broken Performance: Effect of Tokenization on Performance of LLMs
par: Pawar, Sachin, et autres
Publié: (2025)

Comparative Analysis of Tokenization Algorithms for Low-Resource Language Dzongkha
par: Wangchuk, Tandin, et autres
Publié: (2025)

Tokenizer Fertility and Zero-Shot Performance of Foundation Models on Ukrainian Legal Text: A Comparative Study
par: Ovcharov, Volodymyr
Publié: (2026)

From Retrieval to Generation: Comparing Different Approaches
par: Abdallah, Abdelrahman, et autres
Publié: (2025)

Estimating Agreement by Chance for Sequence Annotation
par: Li, Diya, et autres
Publié: (2024)

Lost in Variation? Evaluating NLI Performance in Basque and Spanish Geographical Variants
par: Bengoetxea, Jaione, et autres
Publié: (2025)

Comparing Apples to Oranges: A Dataset & Analysis of LLM Humour Understanding from Traditional Puns to Topical Jokes
par: Loakman, Tyler, et autres
Publié: (2025)

Regress, Don't Guess -- A Regression-like Loss on Number Tokens for Language Models
par: Zausinger, Jonas, et autres
Publié: (2024)

Tokenization Multiplicity Leads to Arbitrary Price Variation in LLM-as-a-service
par: Chatzi, Ivi, et autres
Publié: (2025)

RSQ: Learning from Important Tokens Leads to Better Quantized LLMs
par: Sung, Yi-Lin, et autres
Publié: (2025)

Characterizing Learning Curves During Language Model Pre-Training: Learning, Forgetting, and Stability
par: Chang, Tyler A., et autres
Publié: (2023)

FoNE: Precise Single-Token Number Embeddings via Fourier Features
par: Zhou, Tianyi, et autres
Publié: (2025)

How Do Transformers Learn to Associate Tokens: Gradient Leading Terms Bring Mechanistic Interpretability
par: Im, Shawn, et autres
Publié: (2026)