Cover Image

Roll the dice & look before you leap: Going beyond the creative limits of next-token prediction

Fuente: arXiv

Saved in:

Bibliographic Details
Main Authors:	Nagarajan, Vaishnavh, Wu, Chen Henry, Ding, Charles, Raghunathan, Aditi
Format:	Preprint
Published:	2025
Subjects:	Machine Learning Artificial Intelligence Computation and Language
Online Access:	Acceder al recurso
Tags:	Add Tag No Tags, Be the first to tag this record!

Similar Items

The pitfalls of next-token prediction
by: Bachmann, Gregor, et al.
Published: (2024)

Think before you speak: Training Language Models With Pause Tokens
by: Goyal, Sachin, et al.
Published: (2023)

Looking beyond the next token
by: Thankaraj, Abitha, et al.
Published: (2025)

Self-Trained Verification for Training- and Test-Time Self-Improvement
by: Wu, Chen Henry, et al.
Published: (2026)

Mode-Conditioning Unlocks Superior Test-Time Scaling
by: Wu, Chen Henry, et al.
Published: (2025)

Language models are better than humans at next-token prediction
by: Shlegeris, Buck, et al.
Published: (2022)

Deep sequence models tend to memorize geometrically; it is unclear why
by: Noroozizadeh, Shahriar, et al.
Published: (2025)

Catch Your Breath: Adaptive Computation for Self-Paced Sequence Production
by: Galashov, Alexandre, et al.
Published: (2025)

Sharpness-Aware Minimization Enhances Feature Quality via Balanced Learning
by: Springer, Jacob Mitchell, et al.
Published: (2024)

Scaling Transformer to 1M tokens and beyond with RMT
by: Bulatov, Aydar, et al.
Published: (2023)

AnomaLLMy -- Detecting anomalous tokens in black-box LLMs through low-confidence single-token predictions
by: Witold, Waligóra
Published: (2024)

Mitigating Modal Imbalance in Multimodal Reasoning
by: Wu, Chen Henry, et al.
Published: (2025)

Jailbreaking in the Haystack
by: Shah, Rishi Rajesh, et al.
Published: (2025)

Base Models Look Human To AI Detectors
by: Xu, Yixuan Even, et al.
Published: (2026)

Prediction hubs are context-informed frequent tokens in LLMs
by: Nielsen, Beatrix M. G., et al.
Published: (2025)

Jacobian Scopes: token-level causal attributions in LLMs
by: Liu, Toni J. B., et al.
Published: (2026)

Overtrained Language Models Are Harder to Fine-Tune
by: Springer, Jacob Mitchell, et al.
Published: (2025)

On the token distance modeling ability of higher RoPE attention dimension
by: Hong, Xiangyu, et al.
Published: (2024)

Prompting Science Report 3: I'll pay you or I'll kill you -- but will you care?
by: Meincke, Lennart, et al.
Published: (2025)

Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context
by: Gemini Team, et al.
Published: (2024)

Large-scale cloze evaluation reveals that token prediction tasks are neither lexically nor semantically aligned
by: Jacobs, Cassandra L., et al.
Published: (2024)

On student-teacher deviations in distillation: does it pay to disobey?
by: Nagarajan, Vaishnavh, et al.
Published: (2023)

Beyond checkmate: exploring the creative chokepoints in AI text
by: Tripto, Nafis Irtiza, et al.
Published: (2025)

Shaping capabilities with token-level data filtering
by: Rathi, Neil, et al.
Published: (2026)

Flow Map Language Models: One-step Language Modeling via Continuous Denoising
by: Lee, Chanhyuk, et al.
Published: (2026)

Scaled and Inter-token Relation Enhanced Transformer for Sample-restricted Residential NILM
by: Rahman, Minhajur, et al.
Published: (2024)

Multi-agent AI systems outperform human teams in creativity
by: Hu, Tiancheng, et al.
Published: (2026)

DeepMLF: Multimodal language model with learnable tokens for deep fusion in sentiment analysis
by: Georgiou, Efthymios, et al.
Published: (2025)

Towards Nepali-language LLMs: Efficient GPT training with a Nepali BPE tokenizer
by: Shrestha, Adarsha, et al.
Published: (2025)

SubTokenTest: A Practical Benchmark for Real-World Sub-token Understanding
by: Hou, Shuyang, et al.
Published: (2026)

Hodoscope: Unsupervised Monitoring for AI Misbehaviors
by: Zhong, Ziqian, et al.
Published: (2026)

Not all tokens are created equal: Perplexity Attention Weighted Networks for AI generated text detection
by: Miralles-González, Pablo, et al.
Published: (2025)

Not-Just-Scaling Laws: Towards a Better Understanding of the Downstream Impact of Language Model Design Decisions
by: Liu, Emmy, et al.
Published: (2025)

Interpretable Next-token Prediction via the Generalized Induction Head
by: Kim, Eunji, et al.
Published: (2024)

Forma mentis networks predict creativity ratings of short texts via interpretable artificial intelligence in human and GPT-simulated raters
by: Haim, Edith, et al.
Published: (2024)

COMPACT: Common-token Optimized Model Pruning Across Channels and Tokens
by: Kwek, Eugene, et al.
Published: (2025)

Unused information in token probability distribution of generative LLM: improving LLM reading comprehension through calculation of expected values
by: Zawistowski, Krystian
Published: (2024)

Sudoku-Bench: Evaluating creative reasoning with Sudoku variants
by: Seely, Jeffrey, et al.
Published: (2025)

All or None: Identifiable Linear Properties of Next-token Predictors in Language Modeling
by: Marconato, Emanuele, et al.
Published: (2024)

Essential-Web v1.0: 24T tokens of organized web data
by: AI, Essential, et al.
Published: (2025)