Cover Image

Multi-Token Prediction via Self-Distillation

Fuente: arXiv

Saved in:

Bibliographic Details
Main Authors:	Kirchenbauer, John, Hans, Abhimanyu, Bartoldson, Brian, Goldblum, Micah, Panda, Ashwinee, Goldstein, Tom
Format:	Preprint
Published:	2026
Subjects:	Computation and Language Machine Learning
Online Access:	Acceder al recurso
Tags:	Add Tag No Tags, Be the first to tag this record!

Similar Items

Refusal Tokens: A Simple Way to Calibrate Refusals in Large Language Models
by: Jain, Neel, et al.
Published: (2024)

Teaching Pretrained Language Models to Think Deeper with Retrofitted Recurrence
by: McLeish, Sean, et al.
Published: (2025)

LoRI: Reducing Cross-Task Interference in Multi-Task Low-Rank Adaptation
by: Zhang, Juzheng, et al.
Published: (2025)

Gemstones: A Model Suite for Multi-Faceted Scaling Laws
by: McLeish, Sean, et al.
Published: (2025)

Spotting LLMs With Binoculars: Zero-Shot Detection of Machine-Generated Text
by: Hans, Abhimanyu, et al.
Published: (2024)

Scaling up Test-Time Compute with Latent Reasoning: A Recurrent Depth Approach
by: Geiping, Jonas, et al.
Published: (2025)

On the Reliability of Watermarks for Large Language Models
by: Kirchenbauer, John, et al.
Published: (2023)

FictionalQA: A Dataset for Studying Memorization and Knowledge Acquisition
by: Kirchenbauer, John, et al.
Published: (2025)

Identifying and Evaluating Inactive Heads in Pretrained LLMs
by: Sandoval-Segura, Pedro, et al.
Published: (2025)

GATES: Self-Distillation under Privileged Context with Consensus Gating
by: Stein, Alex, et al.
Published: (2026)

Self-Distillation for Multi-Token Prediction
by: Zhao, Guoliang, et al.
Published: (2026)

A Watermark for Large Language Models
by: Kirchenbauer, John, et al.
Published: (2023)

LMD3: Language Model Data Density Dependence
by: Kirchenbauer, John, et al.
Published: (2024)

OPTune: Efficient Online Preference Tuning
by: Chen, Lichang, et al.
Published: (2024)

DynaGuard: A Dynamic Guardian Model With User-Defined Policies
by: Hoover, Monte, et al.
Published: (2025)

Antidistillation Fingerprinting
by: Xu, Yixuan Even, et al.
Published: (2026)

FineGRAIN: Evaluating Failure Modes of Text-to-Image Models with Vision Language Model Judges
by: Hayes, Kevin David, et al.
Published: (2025)

Private Fine-tuning of Large Language Models with Zeroth-order Optimization
by: Tang, Xinyu, et al.
Published: (2024)

A Simple Baseline for Predicting Events with Auto-Regressive Tabular Transformers
by: Stein, Alex, et al.
Published: (2024)

Zebra-CoT: A Dataset for Interleaved Vision Language Reasoning
by: Li, Ang, et al.
Published: (2025)

The Common Pile v0.1: An 8TB Dataset of Public Domain and Openly Licensed Text
by: Kandpal, Nikhil, et al.
Published: (2025)

X-Token: Projection-Guided Cross-Tokenizer Knowledge Distillation
by: Sreenivas, Sharath Turuvekere, et al.
Published: (2026)

Token Distillation: Attention-aware Input Embeddings For New Tokens
by: Dobler, Konstantin, et al.
Published: (2025)

Transformers Boost the Performance of Decision Trees on Tabular Data across Sample Sizes
by: Jayawardhana, Mayuka, et al.
Published: (2025)

$π$-Play: Multi-Agent Self-Play via Privileged Self-Distillation without External Data
by: Zhang, Yaocheng, et al.
Published: (2026)

Looking Beyond The Top-1: Transformers Determine Top Tokens In Order
by: Lioubashevski, Daria, et al.
Published: (2024)

Speculative Diffusion Decoding: Accelerating Language Generation through Diffusion
by: Christopher, Jacob K, et al.
Published: (2024)

Speculating Experts Accelerates Inference for Mixture-of-Experts
by: Madan, Vivan, et al.
Published: (2026)

LiveMathematicianBench: A Live Benchmark for Mathematician-Level Reasoning with Proof Sketches
by: He, Linyang, et al.
Published: (2026)

Reasoning Bias of Next Token Prediction Training
by: Lin, Pengxiao, et al.
Published: (2025)

Teach LLMs to Phish: Stealing Private Information from Language Models
by: Panda, Ashwinee, et al.
Published: (2024)

Efficient Knowledge Injection in LLMs via Self-Distillation
by: Kujanpää, Kalle, et al.
Published: (2024)

Commercial LLM Agents Are Already Vulnerable to Simple Yet Dangerous Attacks
by: Li, Ang, et al.
Published: (2025)

Self-Distilled Reasoner: On-Policy Self-Distillation for Large Language Models
by: Zhao, Siyan, et al.
Published: (2026)

Self-Distilled RLVR
by: Yang, Chenxu, et al.
Published: (2026)

Alignment-Constrained Dynamic Pruning for LLMs: Identifying and Preserving Alignment-Critical Circuits
by: Patel, Dev, et al.
Published: (2025)

Self-Refining Language Model Anonymizers via Adversarial Distillation
by: Kim, Kyuyoung, et al.
Published: (2025)

Cross-Tokenizer Likelihood Scoring Algorithms for Language Model Distillation
by: Phan, Buu, et al.
Published: (2025)

Probe and Skip: Self-Predictive Token Skipping for Efficient Long-Context LLM Inference
by: Wu, Zimeng, et al.
Published: (2026)

Self-Calibrating Language Models via Test-Time Discriminative Distillation
by: Hedna, Mohamed Rissal, et al.
Published: (2026)