Cover Image

Tokenization Matters: Navigating Data-Scarce Tokenization for Gender Inclusive Language Technologies

Fuente: arXiv

Saved in:

Bibliographic Details
Main Authors:	Ovalle, Anaelia, Mehrabi, Ninareh, Goyal, Palash, Dhamala, Jwala, Chang, Kai-Wei, Zemel, Richard, Galstyan, Aram, Pinter, Yuval, Gupta, Rahul
Format:	Preprint
Published:	2023
Subjects:	Computation and Language Artificial Intelligence Machine Learning
Online Access:	Acceder al recurso
Tags:	Add Tag No Tags, Be the first to tag this record!

Similar Items

Tree-of-Traversals: A Zero-Shot Reasoning Algorithm for Augmenting Black-box Language Models with Knowledge Graphs
by: Markowitz, Elan, et al.
Published: (2024)

On the steerability of large language models toward data-driven personas
by: Li, Junyi, et al.
Published: (2023)

Data Advisor: Dynamic Data Curation for Safety Alignment of Large Language Models
by: Wang, Fei, et al.
Published: (2024)

Attribute Controlled Fine-tuning for Large Language Models: A Case Study on Detoxification
by: Meng, Tao, et al.
Published: (2024)

FLIRT: Feedback Loop In-context Red Teaming
by: Mehrabi, Ninareh, et al.
Published: (2023)

Kaleidoscopic Teaming in Multi Agent Simulations
by: Mehrabi, Ninareh, et al.
Published: (2025)

Towards Safety Reasoning in LLMs: AI-agentic Deliberation for Policy-embedded CoT Data Creation
by: Kumarage, Tharindu, et al.
Published: (2025)

K-Edit: Language Model Editing with Contextual Knowledge Awareness
by: Markowitz, Elan, et al.
Published: (2025)

DECOR: Auditing LLM Deception via Information Manipulation Theory
by: Cai, Linyue, et al.
Published: (2026)

Diagnosing Memorization in Chain-of-Thought Reasoning, One Token at a Time
by: Li, Huihan, et al.
Published: (2025)

SWAN: Semantic Watermarking with Abstract Meaning Representation
by: Ye, Ziping, et al.
Published: (2026)

Which Pieces Does Unigram Tokenization Really Need?
by: Land, Sander, et al.
Published: (2025)

CharBench: Evaluating the Role of Tokenization in Character-Level Tasks
by: Uzan, Omri, et al.
Published: (2025)

Faster Superword Tokenization
by: Schmidt, Craig W., et al.
Published: (2026)

Protecting Privacy in Classifiers by Token Manipulation
by: Harel, Re'em, et al.
Published: (2024)

Prompt Perturbation Consistency Learning for Robust Language Models
by: Qiang, Yao, et al.
Published: (2024)

Token-Level Privacy in Large Language Models
by: Harel, Re'em, et al.
Published: (2025)

Splintering Nonconcatenative Languages for Better Tokenization
by: Gazit, Bar, et al.
Published: (2025)

LH-Deception: Simulating and Understanding LLM Deceptive Behaviors in Long-Horizon Interactions
by: Xu, Yang, et al.
Published: (2025)

Faithful Model Evaluation for Model-Based Metrics
by: Goyal, Palash, et al.
Published: (2023)

How Much is Enough? The Diminishing Returns of Tokenization Training Data
by: Reddy, Varshini, et al.
Published: (2025)

Greed is All You Need: An Evaluation of Tokenizer Inference Methods
by: Uzan, Omri, et al.
Published: (2024)

Tokenization with Split Trees
by: Schmidt, Craig W., et al.
Published: (2026)

Tokenization Is More Than Compression
by: Schmidt, Craig W., et al.
Published: (2024)

Strategize Globally, Adapt Locally: A Multi-Turn Red Teaming Agent with Dual-Level Learning
by: Chen, Si, et al.
Published: (2025)

DiCoRe: Enhancing Zero-shot Event Detection via Divergent-Convergent LLM Reasoning
by: Parekh, Tanmay, et al.
Published: (2025)

Robust Persona-Aware Toxicity Detection with Prompt Optimization and Learned Ensembling
by: Atil, Berk, et al.
Published: (2026)

Evaluating Subword Tokenization: Alien Subword Composition and OOV Generalization Challenge
by: Batsuren, Khuyagbaatar, et al.
Published: (2024)

Evaluating the Critical Risks of Amazon's Nova Premier under the Frontier Model Safety Framework
by: Krishna, Satyapriya, et al.
Published: (2025)

RoPE Distinguishes Neither Positions Nor Tokens in Long Contexts, Provably
by: Du, Yufeng, et al.
Published: (2026)

Broken-Token: Filtering Obfuscated Prompts by Counting Characters-Per-Token
by: Zychlinski, Shaked, et al.
Published: (2025)

The Root Shapes the Fruit: On the Persistence of Gender-Exclusive Harms in Aligned Language Models
by: Ovalle, Anaelia, et al.
Published: (2024)

ARES: Adaptive Red-Teaming and End-to-End Repair of Policy-Reward System
by: Liang, Jiacheng, et al.
Published: (2026)

Asymmetric Phase Coding Audio Watermarking
by: Yang, Guang, et al.
Published: (2026)

FERRET: Framework for Expansion Reliant Red Teaming
by: Mehrabi, Ninareh, et al.
Published: (2026)

Think Clearly: Improving Reasoning via Redundant Token Pruning
by: Choi, Daewon, et al.
Published: (2025)

Not Every Token Needs Forgetting: Selective Unlearning to Limit Change in Utility in Large Language Model Unlearning
by: Wan, Yixin, et al.
Published: (2025)

Probing Subphonemes in Morphology Models
by: Astrach, Gal, et al.
Published: (2025)

Hebrew Diacritics Restoration using Visual Representation
by: Elboher, Yair, et al.
Published: (2025)

Don't Touch My Diacritics
by: Gorman, Kyle, et al.
Published: (2024)