Image de couverture de livre

On Mesa-Optimization in Autoregressively Trained Transformers: Emergence and Capability

Fuente: arXiv

Enregistré dans:

Détails bibliographiques
Auteurs principaux:	Zheng, Chenyu, Huang, Wei, Wang, Rongzhen, Wu, Guoqiang, Zhu, Jun, Li, Chongxuan
Format:	Preprint
Publié:	2024
Sujets:	Machine Learning Computation and Language
Accès en ligne:	Acceder al recurso
Tags:	Ajouter un tag Pas de tags, Soyez le premier à ajouter un tag!

Documents similaires

A Theory for Conditional Generative Modeling on Multiple Data Sources
par: Wang, Rongzhen, et autres
Publié: (2025)

Scaling Diffusion Transformers Efficiently via $μ$P
par: Zheng, Chenyu, et autres
Publié: (2025)

ReFusion: A Diffusion Large Language Model with Parallel Autoregressive Decoding
par: Li, Jia-Nan, et autres
Publié: (2025)

Spectral Condition for $μ$P under Width-Depth Scaling
par: Zheng, Chenyu, et autres
Publié: (2026)

Evolving LLMs' Self-Refinement Capability via Synergistic Training-Inference Optimization
par: Zeng, Yongcheng, et autres
Publié: (2025)

ALPINE: Unveiling the Planning Capability of Autoregressive Learning in Language Models
par: Wang, Siwei, et autres
Publié: (2024)

Make Some Noise: Unlocking Language Model Parallel Inference Capability through Noisy Training
par: Wang, Yixuan, et autres
Publié: (2024)

Flexible Language Modeling in Continuous Space with Transformer-based Autoregressive Flows
par: Zhang, Ruixiang, et autres
Publié: (2025)

Capability-Oriented Training Induced Alignment Risk
par: Zhou, Yujun, et autres
Publié: (2026)

Cognitive Fatigue in Autoregressive Transformers: Formalization and Measurement
par: Marwah, Riju, et autres
Publié: (2026)

How does Multi-Task Training Affect Transformer In-Context Capabilities? Investigations with Function Classes
par: Bhasin, Harmon, et autres
Publié: (2024)

Improving Autoregressive Training with Dynamic Oracles
par: Yang, Jianing, et autres
Publié: (2024)

Your Absorbing Discrete Diffusion Secretly Models the Conditional Distributions of Clean Data
par: Ou, Jingyang, et autres
Publié: (2024)

Ultra Memory-Efficient On-FPGA Training of Transformers via Tensor-Compressed Optimization
par: Tian, Jiayi, et autres
Publié: (2025)

ORLM: A Customizable Framework in Training Large Models for Automated Optimization Modeling
par: Huang, Chenyu, et autres
Publié: (2024)

Dynamic Context Pruning for Efficient and Interpretable Autoregressive Transformers
par: Anagnostidis, Sotiris, et autres
Publié: (2023)

Large Language Diffusion Models
par: Nie, Shen, et autres
Publié: (2025)

Position: The Turing-Completeness of Autoregressive Transformers Relies Heavily on Context Management
par: Cui, Guanyu, et autres
Publié: (2026)

MesaNet: Sequence Modeling by Locally Optimal Test-Time Training
par: von Oswald, Johannes, et autres
Publié: (2025)

On the Limited Generalization Capability of the Implicit Reward Model Induced by Direct Preference Optimization
par: Lin, Yong, et autres
Publié: (2024)

Differential Transformer
par: Ye, Tianzhu, et autres
Publié: (2024)

CAST: Continuous and Differentiable Semi-Structured Sparsity-Aware Training for Large Language Models
par: Huang, Weiyu, et autres
Publié: (2025)

Beyond Autoregression: Discrete Diffusion for Complex Reasoning and Planning
par: Ye, Jiacheng, et autres
Publié: (2024)

Holistic Capability Preservation: Towards Compact Yet Comprehensive Reasoning Models
par: Ling Team, et autres
Publié: (2025)

Mask-Enhanced Autoregressive Prediction: Pay Less Attention to Learn More
par: Zhuang, Xialie, et autres
Publié: (2025)

Why Are Positional Encodings Nonessential for Deep Autoregressive Transformers? Revisiting a Petroglyph
par: Irie, Kazuki
Publié: (2024)

Latent Recurrent Transformer: Architecture Exploration, Training Strategies, and Scaling Behavior
par: Huang, Zeyi, et autres
Publié: (2026)

Momentum Streams for Optimizer-Inspired Transformers
par: Gai, Jingchu, et autres
Publié: (2026)

Trained on Tokens, Calibrated on Concepts: The Emergence of Semantic Calibration in LLMs
par: Nakkiran, Preetum, et autres
Publié: (2025)

You Only Need Minimal RLVR Training: Extrapolating LLMs via Rank-1 Trajectories
par: Wei, Zhepei, et autres
Publié: (2026)

LeaPformer: Enabling Linear Transformers for Autoregressive and Simultaneous Tasks via Learned Proportions
par: Agostinelli, Victor, et autres
Publié: (2024)

Algorithmic Capabilities of Random Transformers
par: Zhong, Ziqian, et autres
Publié: (2024)

Look Ahead or Look Around? A Theoretical Comparison Between Autoregressive and Masked Pretraining
par: Zhang, Qi, et autres
Publié: (2024)

PoSE: Efficient Context Window Extension of LLMs via Positional Skip-wise Training
par: Zhu, Dawei, et autres
Publié: (2023)

Mechanism and Emergence of Stacked Attention Heads in Multi-Layer Transformers
par: Musat, Tiberiu
Publié: (2024)

Advancing Sequential Numerical Prediction in Autoregressive Models
par: Fei, Xiang, et autres
Publié: (2025)

Emergence of Episodic Memory in Transformers: Characterizing Changes in Temporal Structure of Attention Scores During Training
par: Mistry, Deven Mahesh, et autres
Publié: (2025)

AutoTimes: Autoregressive Time Series Forecasters via Large Language Models
par: Liu, Yong, et autres
Publié: (2024)

Hephaestus: Improving Fundamental Agent Capabilities of Large Language Models through Continual Pre-Training
par: Zhuang, Yuchen, et autres
Publié: (2025)

Universal Approximation of Visual Autoregressive Transformers
par: Chen, Yifang, et autres
Publié: (2025)