Image de couverture de livre

Repeat After Me: Transformers are Better than State Space Models at Copying

Fuente: arXiv

Enregistré dans:

Détails bibliographiques
Auteurs principaux:	Jelassi, Samy, Brandfonbrener, David, Kakade, Sham M., Malach, Eran
Format:	Preprint
Publié:	2024
Sujets:	Machine Learning Artificial Intelligence Computation and Language
Accès en ligne:	Acceder al recurso
Tags:	Ajouter un tag Pas de tags, Soyez le premier à ajouter un tag!

Documents similaires

The Role of Sparsity for Length Generalization in Transformers
par: Golowich, Noah, et autres
Publié: (2025)

Loss-to-Loss Prediction: Scaling Laws for All Datasets
par: Brandfonbrener, David, et autres
Publié: (2024)

Universal Length Generalization with Turing Programs
par: Hou, Kaiying, et autres
Publié: (2024)

Let Me Think! A Long Chain-of-Thought Can Be Worth Exponentially Many Short Ones
par: Mirtaheri, Parsa, et autres
Publié: (2025)

LoRA Soups: Merging LoRAs for Practical Skill Composition Tasks
par: Prabhakar, Akshara, et autres
Publié: (2024)

CoLoR-Filter: Conditional Loss Reduction Filtering for Targeted Language Model Pre-training
par: Brandfonbrener, David, et autres
Publié: (2024)

Mixture of Parrots: Experts improve memorization more than reasoning
par: Jelassi, Samy, et autres
Publié: (2024)

Echo Chamber: RL Post-training Amplifies Behaviors Learned in Pretraining
par: Zhao, Rosie, et autres
Publié: (2025)

Q-Probe: A Lightweight Approach to Reward Maximization for Language Models
par: Li, Kenneth, et autres
Publié: (2024)

Deconstructing What Makes a Good Optimizer for Language Models
par: Zhao, Rosie, et autres
Publié: (2024)

Prescriptive Scaling Reveals the Evolution of Language Model Capabilities
par: Zhang, Hanlin, et autres
Publié: (2026)

Discovering Hierarchical Latent Capabilities of Language Models via Causal Representation Learning
par: Jin, Jikai, et autres
Publié: (2025)

Auto-Regressive Next-Token Predictors are Universal Learners
par: Malach, Eran
Publié: (2023)

To Backtrack or Not to Backtrack: When Sequential Search Limits Model Reasoning
par: Qin, Tian, et autres
Publié: (2025)

Transcendence: Generative Models Can Outperform The Experts That Train Them
par: Zhang, Edwin, et autres
Publié: (2024)

Follow My Instruction and Spill the Beans: Scalable Data Extraction from Retrieval-Augmented Generation Systems
par: Qi, Zhenting, et autres
Publié: (2024)

Decomposing Elements of Problem Solving: What "Math" Does RL Teach?
par: Qin, Tian, et autres
Publié: (2025)

EvoLM: In Search of Lost Language Model Training Dynamics
par: Qi, Zhenting, et autres
Publié: (2025)

Language Models "Grok" to Copy
par: Lv, Ang, et autres
Publié: (2024)

Are You Sure? Rank Them Again: Repeated Ranking For Better Preference Datasets
par: Devine, Peter
Publié: (2024)

A Study on the Calibration of In-context Learning
par: Zhang, Hanlin, et autres
Publié: (2023)

SOAP: Improving and Stabilizing Shampoo using Adam
par: Vyas, Nikhil, et autres
Publié: (2024)

Sparse MeZO: Less Parameters for Better Performance in Zeroth-Order LLM Fine-Tuning
par: Liu, Yong, et autres
Publié: (2024)

OptiMer: Optimal Distribution Vector Merging Is Better than Data Mixing for Continual Pre-Training
par: Song, Haiyue, et autres
Publié: (2026)

Language Models can Self-Improve at State-Value Estimation for Better Search
par: Mendes, Ethan, et autres
Publié: (2025)

STAT: Shrinking Transformers After Training
par: Flynn, Megan, et autres
Publié: (2024)

MeMo: Memory as a Model
par: Quek, Ryan Wei Heng, et autres
Publié: (2026)

ReplaceMe: Network Simplification via Depth Pruning and Transformer Block Linearization
par: Shopkhoev, Dmitriy, et autres
Publié: (2025)

The Belief State Transformer
par: Hu, Edward S., et autres
Publié: (2024)

Tracing Facts or just Copies? A critical investigation of the Competitions of Mechanisms in Large Language Models
par: Campregher, Dante, et autres
Publié: (2025)

The Recurrent Transformer: Greater Effective Depth and Efficient Decoding
par: Oncescu, Costin-Andrei, et autres
Publié: (2026)

Learning Hidden Markov Models Using Conditional Samples
par: Kakade, Sham M., et autres
Publié: (2023)

Taipan: Efficient and Expressive State Space Language Models with Selective Attention
par: Van Nguyen, Chien, et autres
Publié: (2024)

MemMamba: Rethinking Memory Patterns in State Space Model
par: Wang, Youjin, et autres
Publié: (2025)

PICASO: Permutation-Invariant Context Composition with State Space Models
par: Liu, Tian Yu, et autres
Publié: (2025)

Technologies on Effectiveness and Efficiency: A Survey of State Spaces Models
par: Lv, Xingtai, et autres
Publié: (2025)

GQ-VAE: A gated quantized VAE for learning variable length tokens
par: Datta, Theo, et autres
Publié: (2025)

On the Expressiveness and Length Generalization of Selective State-Space Models on Regular Languages
par: Terzić, Aleksandar, et autres
Publié: (2024)

Birdie: Advancing State Space Models with Reward-Driven Objectives and Curricula
par: Blouir, Sam, et autres
Publié: (2024)

Instruction-tuned Language Models are Better Knowledge Learners
par: Jiang, Zhengbao, et autres
Publié: (2024)