Imagen de Portada

Tokenization Matters: Navigating Data-Scarce Tokenization for Gender Inclusive Language Technologies

Fuente: arXiv

Guardado en:

Detalles Bibliográficos
Autores principales:	Ovalle, Anaelia, Mehrabi, Ninareh, Goyal, Palash, Dhamala, Jwala, Chang, Kai-Wei, Zemel, Richard, Galstyan, Aram, Pinter, Yuval, Gupta, Rahul
Formato:	Preprint
Publicado:	2023
Materias:	Computation and Language Artificial Intelligence Machine Learning
Acceso en línea:	Acceder al recurso
Etiquetas:	Agregar Etiqueta Sin Etiquetas, Sea el primero en etiquetar este registro!

Ejemplares similares

Tree-of-Traversals: A Zero-Shot Reasoning Algorithm for Augmenting Black-box Language Models with Knowledge Graphs
por: Markowitz, Elan, et al.
Publicado: (2024)

On the steerability of large language models toward data-driven personas
por: Li, Junyi, et al.
Publicado: (2023)

Data Advisor: Dynamic Data Curation for Safety Alignment of Large Language Models
por: Wang, Fei, et al.
Publicado: (2024)

Attribute Controlled Fine-tuning for Large Language Models: A Case Study on Detoxification
por: Meng, Tao, et al.
Publicado: (2024)

FLIRT: Feedback Loop In-context Red Teaming
por: Mehrabi, Ninareh, et al.
Publicado: (2023)

Kaleidoscopic Teaming in Multi Agent Simulations
por: Mehrabi, Ninareh, et al.
Publicado: (2025)

Towards Safety Reasoning in LLMs: AI-agentic Deliberation for Policy-embedded CoT Data Creation
por: Kumarage, Tharindu, et al.
Publicado: (2025)

K-Edit: Language Model Editing with Contextual Knowledge Awareness
por: Markowitz, Elan, et al.
Publicado: (2025)

DECOR: Auditing LLM Deception via Information Manipulation Theory
por: Cai, Linyue, et al.
Publicado: (2026)

Diagnosing Memorization in Chain-of-Thought Reasoning, One Token at a Time
por: Li, Huihan, et al.
Publicado: (2025)

SWAN: Semantic Watermarking with Abstract Meaning Representation
por: Ye, Ziping, et al.
Publicado: (2026)

Which Pieces Does Unigram Tokenization Really Need?
por: Land, Sander, et al.
Publicado: (2025)

CharBench: Evaluating the Role of Tokenization in Character-Level Tasks
por: Uzan, Omri, et al.
Publicado: (2025)

Faster Superword Tokenization
por: Schmidt, Craig W., et al.
Publicado: (2026)

Protecting Privacy in Classifiers by Token Manipulation
por: Harel, Re'em, et al.
Publicado: (2024)

Prompt Perturbation Consistency Learning for Robust Language Models
por: Qiang, Yao, et al.
Publicado: (2024)

Token-Level Privacy in Large Language Models
por: Harel, Re'em, et al.
Publicado: (2025)

Splintering Nonconcatenative Languages for Better Tokenization
por: Gazit, Bar, et al.
Publicado: (2025)

LH-Deception: Simulating and Understanding LLM Deceptive Behaviors in Long-Horizon Interactions
por: Xu, Yang, et al.
Publicado: (2025)

Faithful Model Evaluation for Model-Based Metrics
por: Goyal, Palash, et al.
Publicado: (2023)

How Much is Enough? The Diminishing Returns of Tokenization Training Data
por: Reddy, Varshini, et al.
Publicado: (2025)

Greed is All You Need: An Evaluation of Tokenizer Inference Methods
por: Uzan, Omri, et al.
Publicado: (2024)

Tokenization with Split Trees
por: Schmidt, Craig W., et al.
Publicado: (2026)

Tokenization Is More Than Compression
por: Schmidt, Craig W., et al.
Publicado: (2024)

Strategize Globally, Adapt Locally: A Multi-Turn Red Teaming Agent with Dual-Level Learning
por: Chen, Si, et al.
Publicado: (2025)

DiCoRe: Enhancing Zero-shot Event Detection via Divergent-Convergent LLM Reasoning
por: Parekh, Tanmay, et al.
Publicado: (2025)

Robust Persona-Aware Toxicity Detection with Prompt Optimization and Learned Ensembling
por: Atil, Berk, et al.
Publicado: (2026)

Evaluating Subword Tokenization: Alien Subword Composition and OOV Generalization Challenge
por: Batsuren, Khuyagbaatar, et al.
Publicado: (2024)

Evaluating the Critical Risks of Amazon's Nova Premier under the Frontier Model Safety Framework
por: Krishna, Satyapriya, et al.
Publicado: (2025)

RoPE Distinguishes Neither Positions Nor Tokens in Long Contexts, Provably
por: Du, Yufeng, et al.
Publicado: (2026)

Broken-Token: Filtering Obfuscated Prompts by Counting Characters-Per-Token
por: Zychlinski, Shaked, et al.
Publicado: (2025)

The Root Shapes the Fruit: On the Persistence of Gender-Exclusive Harms in Aligned Language Models
por: Ovalle, Anaelia, et al.
Publicado: (2024)

ARES: Adaptive Red-Teaming and End-to-End Repair of Policy-Reward System
por: Liang, Jiacheng, et al.
Publicado: (2026)

Asymmetric Phase Coding Audio Watermarking
por: Yang, Guang, et al.
Publicado: (2026)

FERRET: Framework for Expansion Reliant Red Teaming
por: Mehrabi, Ninareh, et al.
Publicado: (2026)

Think Clearly: Improving Reasoning via Redundant Token Pruning
por: Choi, Daewon, et al.
Publicado: (2025)

Not Every Token Needs Forgetting: Selective Unlearning to Limit Change in Utility in Large Language Model Unlearning
por: Wan, Yixin, et al.
Publicado: (2025)

Probing Subphonemes in Morphology Models
por: Astrach, Gal, et al.
Publicado: (2025)

Hebrew Diacritics Restoration using Visual Representation
por: Elboher, Yair, et al.
Publicado: (2025)

Don't Touch My Diacritics
por: Gorman, Kyle, et al.
Publicado: (2024)