Copertina

Data Mixture Inference: What do BPE Tokenizers Reveal about their Training Data?

Fuente: arXiv

Salvato in:

Dettagli Bibliografici
Autori principali:	Hayase, Jonathan, Liu, Alisa, Choi, Yejin, Oh, Sewoong, Smith, Noah A.
Natura:	Preprint
Pubblicazione:	2024
Soggetti:	Computation and Language Machine Learning
Accesso online:	Acceder al recurso
Tags:	Aggiungi Tag Nessun Tag, puoi essere il primo ad aggiungerne!!

Documenti analoghi

SuperBPE: Space Travel for Language Models
di: Liu, Alisa, et al.
Pubblicazione: (2025)

Sampling from Your Language Model One Byte at a Time
di: Hayase, Jonathan, et al.
Pubblicazione: (2025)

Are you going to finish that? A Practical Study of the Partial Token Problem
di: Xu, Hao, et al.
Pubblicazione: (2026)

Broken Tokens? Your Language Model can Secretly Handle Non-Canonical Tokenizations
di: Zheng, Brian Siyuan, et al.
Pubblicazione: (2025)

Constructing a BPE Tokenization DFA
di: Berglund, Martin, et al.
Pubblicazione: (2024)

A False Sense of Privacy: Evaluating Textual Data Sanitization Beyond Surface-level Privacy Leakage
di: Xin, Rui, et al.
Pubblicazione: (2025)

Optimizing Pre-Training Data Mixtures with Mixtures of Data Expert Models
di: Belenki, Lior, et al.
Pubblicazione: (2025)

GPUTOK: GPU Accelerated Byte Level BPE Tokenization
di: Kadamba, Venu Gopal, et al.
Pubblicazione: (2026)

What's In My Big Data?
di: Elazar, Yanai, et al.
Pubblicazione: (2023)

PLeaS -- Merging Models with Permutations and Least Squares
di: Nasery, Anshul, et al.
Pubblicazione: (2024)

DailyDilemmas: Revealing Value Preferences of LLMs with Quandaries of Daily Life
di: Chiu, Yu Ying, et al.
Pubblicazione: (2024)

Recycling the Web: A Method to Enhance Pre-training Data Quality and Quantity for Language Models
di: Nguyen, Thao, et al.
Pubblicazione: (2025)

Tokens for Learning, Tokens for Unlearning: Mitigating Membership Inference Attacks in Large Language Models via Dual-Purpose Training
di: Tran, Toan, et al.
Pubblicazione: (2025)

Quantifying Language Models' Sensitivity to Spurious Features in Prompt Design or: How I learned to start worrying about prompt formatting
di: Sclar, Melanie, et al.
Pubblicazione: (2023)

When Incentives Backfire, Data Stops Being Human
di: Santy, Sebastin, et al.
Pubblicazione: (2025)

Dense Training, Sparse Inference: Rethinking Training of Mixture-of-Experts Language Models
di: Pan, Bowen, et al.
Pubblicazione: (2024)

What do Transformers Know about Government?
di: Hou, Jue, et al.
Pubblicazione: (2024)

BlockBPE: Parallel BPE Tokenization
di: You, Amos
Pubblicazione: (2025)

Trust or Escalate: LLM Judges with Provable Guarantees for Human Agreement
di: Jung, Jaehun, et al.
Pubblicazione: (2024)

Is Child-Directed Speech Effective Training Data for Language Models?
di: Feng, Steven Y., et al.
Pubblicazione: (2024)

Pretraining Language Models with Subword Regularization: An Empirical Study of BPE Dropout in Low-Resource NLP
di: Visser, Ruan, et al.
Pubblicazione: (2026)

Frequency Explains the Inverse Correlation of Large Language Models' Size, Training Data Amount, and Surprisal's Fit to Reading Times
di: Oh, Byung-Doh, et al.
Pubblicazione: (2024)

DataDecide: How to Predict Best Pretraining Data with Small Experiments
di: Magnusson, Ian, et al.
Pubblicazione: (2025)

Perturb Your Data: Paraphrase-Guided Training Data Watermarking
di: Shetty, Pranav, et al.
Pubblicazione: (2025)

OrthoRank: Token Selection via Sink Token Orthogonality for Efficient LLM inference
di: Shin, Seungjun, et al.
Pubblicazione: (2025)

Mini-batch Coresets for Memory-efficient Language Model Training on Data Mixtures
di: Nguyen, Dang, et al.
Pubblicazione: (2024)

LLAMAPIE: Proactive In-Ear Conversation Assistants
di: Chen, Tuochao, et al.
Pubblicazione: (2025)

Learning to Reason with Mixture of Tokens
di: Jain, Adit, et al.
Pubblicazione: (2025)

Scaling Laws for Mixture Pretraining Under Data Constraints
di: Sedova, Anastasiia, et al.
Pubblicazione: (2026)

On Training Data Influence of GPT Models
di: Chai, Yekun, et al.
Pubblicazione: (2024)

Sample, Don't Search: Rethinking Test-Time Alignment for Language Models
di: Faria, Gonçalo, et al.
Pubblicazione: (2025)

Domain2Vec: Vectorizing Datasets to Find the Optimal Data Mixture without Training
di: Zhang, Mozhi, et al.
Pubblicazione: (2025)

R&B: Domain Regrouping and Data Mixture Balancing for Efficient Foundation Model Training
di: Ge, Albert, et al.
Pubblicazione: (2025)

Never Train from Scratch: Fair Comparison of Long-Sequence Models Requires Data-Driven Priors
di: Amos, Ido, et al.
Pubblicazione: (2023)

Data Mixing Laws: Optimizing Data Mixtures by Predicting Language Modeling Performance
di: Ye, Jiasheng, et al.
Pubblicazione: (2024)

Token Sparse Attention: Efficient Long-Context Inference with Interleaved Token Selection
di: Jo, Dongwon, et al.
Pubblicazione: (2026)

Proximal Causal Inference With Text Data
di: Chen, Jacob M., et al.
Pubblicazione: (2024)

Sequences of Logits Reveal the Low Rank Structure of Language Models
di: Golowich, Noah, et al.
Pubblicazione: (2025)

Mixture of Tokens: Continuous MoE through Cross-Example Aggregation
di: Antoniak, Szymon, et al.
Pubblicazione: (2023)

Tuning Language Models by Proxy
di: Liu, Alisa, et al.
Pubblicazione: (2024)