Imagen de Portada

Exact Byte-Level Probabilities from Tokenized Language Models for FIM-Tasks and Model Ensembles

Fuente: arXiv

Guardado en:

Detalles Bibliográficos
Autores principales:	Phan, Buu, Amos, Brandon, Gat, Itai, Havasi, Marton, Muckley, Matthew, Ullrich, Karen
Formato:	Preprint
Publicado:	2024
Materias:	Computation and Language Machine Learning
Acceso en línea:	Acceder al recurso
Etiquetas:	Agregar Etiqueta Sin Etiquetas, Sea el primero en etiquetar este registro!

Ejemplares similares

Understanding and Mitigating Tokenization Bias in Language Models
por: Phan, Buu, et al.
Publicado: (2024)

Cross-Tokenizer Likelihood Scoring Algorithms for Language Model Distillation
por: Phan, Buu, et al.
Publicado: (2025)

Edit Flows: Flow Matching with Edit Operations
por: Havasi, Marton, et al.
Publicado: (2025)

Corrector Sampling in Language Models
por: Gat, Itai, et al.
Publicado: (2025)

Set Block Decoding is a Language Model Inference Accelerator
por: Gat, Itai, et al.
Publicado: (2025)

ByteFlow: Language Modeling through Adaptive Byte Compression without a Tokenizer
por: Deng, Chunyuan, et al.
Publicado: (2026)

Channel Simulation and Distributed Compression with Ensemble Rejection Sampling
por: Phan, Buu, et al.
Publicado: (2025)

SpaceByte: Towards Deleting Tokenization from Large Language Modeling
por: Slagle, Kevin
Publicado: (2024)

MambaByte: Token-free Selective State Space Model
por: Wang, Junxiong, et al.
Publicado: (2024)

Kronecker Embeddings: Byte-Level Structured Token Representations for Parameter-Efficient Language Models
por: Shravan, Rohan
Publicado: (2026)

Diverse Concept Proposals for Concept Bottleneck Models
por: Brown, Katrina, et al.
Publicado: (2024)

MrT5: Dynamic Token Merging for Efficient Byte-level Language Models
por: Kallini, Julie, et al.
Publicado: (2024)

Scratchpad Patching: Decoupling Compute from Patch Size in Byte-Level Language Models
por: Zheng, Lin, et al.
Publicado: (2026)

List-Level Distribution Coupling with Applications to Speculative Decoding and Lossy Compression
por: Rowan, Joseph, et al.
Publicado: (2025)

Flow Matching with General Discrete Paths: A Kinetic-Optimal Perspective
por: Shaul, Neta, et al.
Publicado: (2024)

Generator Matching: Generative modeling with arbitrary Markov processes
por: Holderrieth, Peter, et al.
Publicado: (2024)

Lossless Compression of Vector IDs for Approximate Nearest Neighbor Search
por: Severo, Daniel, et al.
Publicado: (2025)

GPUTOK: GPU Accelerated Byte Level BPE Tokenization
por: Kadamba, Venu Gopal, et al.
Publicado: (2026)

Just on Time: Token-Level Early Stopping for Diffusion Language Models
por: Kohut, Zahar, et al.
Publicado: (2026)

Hierarchical Autoregressive Transformers: Combining Byte- and Word-Level Processing for Robust, Adaptable Language Models
por: Neitemeier, Pit, et al.
Publicado: (2025)

Language Models over Canonical Byte-Pair Encodings
por: Vieira, Tim, et al.
Publicado: (2025)

Specialising and Analysing Instruction-Tuned and Byte-Level Language Models for Organic Reaction Prediction
por: Pang, Jiayun, et al.
Publicado: (2024)

Byte-token Enhanced Language Models for Temporal Point Processes Analysis
por: Kong, Quyu, et al.
Publicado: (2025)

Online Intrinsic Rewards for Decision Making Agents from Large Language Model Feedback
por: Zheng, Qinqing, et al.
Publicado: (2024)

Scalable Token-Level Hallucination Detection in Large Language Models
por: Min, Rui, et al.
Publicado: (2026)

Flow Matching Guide and Code
por: Lipman, Yaron, et al.
Publicado: (2024)

Guarantee Regions for Local Explanations
por: Havasi, Marton, et al.
Publicado: (2024)

Textually Pretrained Speech Language Models
por: Hassid, Michael, et al.
Publicado: (2023)

Sampling from Your Language Model One Byte at a Time
por: Hayase, Jonathan, et al.
Publicado: (2025)

Transition Matching: Scalable and Flexible Generative Modeling
por: Shaul, Neta, et al.
Publicado: (2025)

TLPO: Token-Level Policy Optimization for Mitigating Language Confusion in Large Language Models
por: Choo, Jinho, et al.
Publicado: (2026)

Detecting Hallucinations in Large Language Model Generation: A Token Probability Approach
por: Quevedo, Ernesto, et al.
Publicado: (2024)

Language Modeling with Learned Meta-Tokens
por: Shah, Alok N., et al.
Publicado: (2025)

Parallel Token Prediction for Language Models
por: Draxler, Felix, et al.
Publicado: (2025)

AlignDistil: Token-Level Language Model Alignment as Adaptive Policy Distillation
por: Zhang, Songming, et al.
Publicado: (2025)

Large Language Models Struggle in Token-Level Clinical Named Entity Recognition
por: Lu, Qiuhao, et al.
Publicado: (2024)

DP-Fusion: Token-Level Differentially Private Inference for Large Language Models
por: Thareja, Rushil, et al.
Publicado: (2025)

Beyond Next Token Prediction: Patch-Level Training for Large Language Models
por: Shao, Chenze, et al.
Publicado: (2024)

TokUR: Token-Level Uncertainty Estimation for Large Language Model Reasoning
por: Zhang, Tunyu, et al.
Publicado: (2025)

Ensembling Finetuned Language Models for Text Classification
por: Arango, Sebastian Pineda, et al.
Publicado: (2024)