Cover Image

Interpretable Next-token Prediction via the Generalized Induction Head

Fuente: arXiv

Saved in:

Bibliographic Details
Main Authors:	Kim, Eunji, Mantena, Sriya, Yang, Weiwei, Singh, Chandan, Yoon, Sungroh, Gao, Jianfeng
Format:	Preprint
Published:	2024
Subjects:	Computation and Language Artificial Intelligence Machine Learning
Online Access:	Acceder al recurso
Tags:	Add Tag No Tags, Be the first to tag this record!

Similar Items

Rethinking Interpretability in the Era of Large Language Models
by: Singh, Chandan, et al.
Published: (2024)

Agentic-imodels: Evolving agentic interpretability tools via autoresearch
by: Singh, Chandan, et al.
Published: (2026)

Large Language Models are Skeptics: False Negative Problem of Input-conflicting Hallucination
by: Song, Jongyoon, et al.
Published: (2024)

Physics in Next-token Prediction
by: An, Hongjun, et al.
Published: (2024)

Learning a Decision Tree Algorithm with Transformers
by: Zhuang, Yufan, et al.
Published: (2024)

Semantic Token Reweighting for Interpretable and Controllable Text Embeddings in CLIP
by: Kim, Eunji, et al.
Published: (2024)

Towards Consistent Natural-Language Explanations via Explanation-Consistency Finetuning
by: Chen, Yanda, et al.
Published: (2024)

Crafting Interpretable Embeddings by Asking LLMs Questions
by: Benara, Vinamra, et al.
Published: (2024)

All or None: Identifiable Linear Properties of Next-token Predictors in Language Modeling
by: Marconato, Emanuele, et al.
Published: (2024)

Self-Supervised Time-Series Anomaly Detection Using Learnable Data Augmentation
by: Choi, Kukjin, et al.
Published: (2024)

R2R2: Robust Representation for Intensive Experience Reuse via Redundancy Reduction in Self-Predictive Learning
by: Song, Sanghyeob, et al.
Published: (2026)

Function Induction and Task Generalization: An Interpretability Study with Off-by-One Addition
by: Ye, Qinyuan, et al.
Published: (2025)

Battling the Non-stationarity in Time Series Forecasting via Test-time Adaptation
by: Kim, HyunGi, et al.
Published: (2025)

Looking beyond the next token
by: Thankaraj, Abitha, et al.
Published: (2025)

The pitfalls of next-token prediction
by: Bachmann, Gregor, et al.
Published: (2024)

Unveiling Induction Heads: Provable Training Dynamics and Feature Learning in Transformers
by: Chen, Siyu, et al.
Published: (2024)

Learning to Interpret Weight Differences in Language Models
by: Goel, Avichal, et al.
Published: (2025)

SUN: Shared Use of Next-token Prediction for Efficient Multi-LLM Disaggregated Serving
by: Woo, Sunghyeon, et al.
Published: (2026)

DefectFill: Realistic Defect Generation with Inpainting Diffusion Model for Visual Inspection
by: Song, Jaewoo, et al.
Published: (2025)

NextLocLLM: Location Semantics Modeling and Coordinate-Based Next Location Prediction with LLMs
by: Liu, Shuai, et al.
Published: (2024)

NewsInterview: a Dataset and a Playground to Evaluate LLMs' Ground Gap via Informational Interviews
by: Spangher, Alexander, et al.
Published: (2024)

Shaping capabilities with token-level data filtering
by: Rathi, Neil, et al.
Published: (2026)

Next-token pretraining implies in-context learning
by: Riechers, Paul M., et al.
Published: (2025)

Entity-level Factual Adaptiveness of Fine-tuning based Abstractive Summarization Models
by: Song, Jongyoon, et al.
Published: (2024)

Scaling Transformer to 1M tokens and beyond with RMT
by: Bulatov, Aydar, et al.
Published: (2023)

Stream Aligner: Efficient Sentence-Level Alignment via Distribution Induction
by: Lou, Hantao, et al.
Published: (2025)

Cautious Next Token Prediction
by: Wang, Yizhou, et al.
Published: (2025)

Language models are better than humans at next-token prediction
by: Shlegeris, Buck, et al.
Published: (2022)

Sample-efficient Adversarial Imitation Learning
by: Jung, Dahuin, et al.
Published: (2023)

Mechanics of Next Token Prediction with Self-Attention
by: Li, Yingcong, et al.
Published: (2024)

COMPACT: Common-token Optimized Model Pruning Across Channels and Tokens
by: Kwek, Eugene, et al.
Published: (2025)

DiZiNER: Disagreement-guided Instruction Refinement via Pilot Annotation Simulation for Zero-shot Named Entity Recognition
by: Kim, Siun, et al.
Published: (2026)

Essential-Web v1.0: 24T tokens of organized web data
by: AI, Essential, et al.
Published: (2025)

A Comprehensive Survey of Deep Learning for Time Series Forecasting: Architectural Diversity and Open Challenges
by: Kim, Jongseon, et al.
Published: (2024)

Interpreting the Effects of Quantization on LLMs
by: Singh, Manpreet, et al.
Published: (2025)

Generating Plausible Distractors for Multiple-Choice Questions via Student Choice Prediction
by: Lee, Yooseop, et al.
Published: (2025)

You only need 4 extra tokens: Synergistic Test-time Adaptation for LLMs
by: Xu, Yijie, et al.
Published: (2025)

Beyond Next-Token Prediction: A Performance Characterization of Diffusion versus Autoregressive Language Models
by: Kim, Minseo, et al.
Published: (2025)

ReEval: Automatic Hallucination Evaluation for Retrieval-Augmented Large Language Models via Transferable Adversarial Attacks
by: Yu, Xiaodong, et al.
Published: (2023)

Interpretable Discriminative Text Representations via Agreement and Label Disentanglement
by: Wang, Tong, et al.
Published: (2026)