Imagen de Portada

Transformers Pretrained on Procedural Data Contain Modular Structures for Algorithmic Reasoning

Fuente: arXiv

Guardado en:

Detalles Bibliográficos
Autores principales:	Shinnick, Zachary, Jiang, Liangze, Saratchandran, Hemanth, Hengel, Anton van den, Teney, Damien
Formato:	Preprint
Publicado:	2025
Materias:	Machine Learning
Acceso en línea:	Acceder al recurso
Etiquetas:	Agregar Etiqueta Sin Etiquetas, Sea el primero en etiquetar este registro!

Ejemplares similares

Procedural Pretraining: Warming Up Language Models with Abstract Data
por: Jiang, Liangze, et al.
Publicado: (2026)

Can You Learn to See Without Images? Procedural Warm-Up for Vision Transformers
por: Shinnick, Zachary, et al.
Publicado: (2025)

OOD-Chameleon: Is Algorithm Selection for OOD Generalization Learnable?
por: Jiang, Liangze, et al.
Publicado: (2024)

Leaner Transformers: More Heads, Less Depth
por: Saratchandran, Hemanth, et al.
Publicado: (2025)

Preconditioned Attention: Enhancing Efficiency in Transformers
por: Saratchandran, Hemanth
Publicado: (2026)

Do We Always Need the Simplicity Bias? Looking for Optimal Inductive Biases in the Wild
por: Teney, Damien, et al.
Publicado: (2025)

Towards Higher Effective Rank in Parameter-efficient Fine-tuning using Khatri--Rao Product
por: Albert, Paul, et al.
Publicado: (2025)

Meta-RL Induces Exploration in Language Agents
por: Jiang, Yulun, et al.
Publicado: (2025)

Spectral Conditioning of Attention Improves Transformer Performance
por: Saratchandran, Hemanth, et al.
Publicado: (2026)

Synergy and Diversity in CLIP: Enhancing Performance Through Adaptive Backbone Ensembling
por: Rodriguez-Opazo, Cristian, et al.
Publicado: (2024)

A Symbolic Framework for Evaluating Mathematical Reasoning and Generalisation with Transformers
por: Meadows, Jordan, et al.
Publicado: (2023)

The Inlet Rank Collapse in Implicit Neural Representations: Diagnosis and Unified Remedy
por: Zheng, Jianqiao, et al.
Publicado: (2026)

The Quantization Benefits of Residual-Free Transformers
por: Ji, Yiping, et al.
Publicado: (2026)

Data Denoising and Derivative Estimation for Data-Driven Modeling of Nonlinear Dynamical Systems
por: Yao, Jiaqi, et al.
Publicado: (2025)

Analyzing the Neural Tangent Kernel of Periodically Activated Coordinate Networks
por: Saratchandran, Hemanth, et al.
Publicado: (2024)

Architectural Strategies for the optimization of Physics-Informed Neural Networks
por: Saratchandran, Hemanth, et al.
Publicado: (2024)

From Activation to Initialization: Scaling Insights for Optimizing Neural Fields
por: Saratchandran, Hemanth, et al.
Publicado: (2024)

Rethinking Attention: Polynomial Alternatives to Softmax in Transformers
por: Saratchandran, Hemanth, et al.
Publicado: (2024)

Confident Sinkhorn Allocation for Pseudo-Labeling
por: Nguyen, Vu, et al.
Publicado: (2022)

Preconditioners for the Stochastic Training of Neural Fields
por: Chng, Shin-Fang, et al.
Publicado: (2024)

Stable Forgetting: Bounded Parameter-Efficient Unlearning in Foundation Models
por: Garg, Arpit, et al.
Publicado: (2025)

Premonition: Using Generative Models to Preempt Future Data Changes in Continual Learning
por: McDonnell, Mark D., et al.
Publicado: (2024)

A Sampling Theory Perspective on Activations for Implicit Neural Representations
por: Saratchandran, Hemanth, et al.
Publicado: (2024)

Always Skip Attention
por: Ji, Yiping, et al.
Publicado: (2025)

Cutting the Skip: Training Residual-Free Transformers
por: Ji, Yiping, et al.
Publicado: (2025)

Concept Component Analysis: A Principled Approach for Concept Extraction in LLMs
por: Liu, Yuhang, et al.
Publicado: (2026)

SineLoRA$Δ$: Sine-Activated Delta Compression
por: Gordon, Cameron, et al.
Publicado: (2025)

Provably Efficient Bayesian Optimization with Unknown Gaussian Process Hyperparameter Estimation
por: Ha, Huong, et al.
Publicado: (2023)

From Tables to Signals: Revealing Spectral Adaptivity in TabPFN
por: Zheng, Jianqiao, et al.
Publicado: (2025)

D'OH: Decoder-Only Random Hypernetworks for Implicit Neural Representations
por: Gordon, Cameron, et al.
Publicado: (2024)

Procedural Knowledge in Pretraining Drives Reasoning in Large Language Models
por: Ruis, Laura, et al.
Publicado: (2024)

Neural Redshift: Random Networks are not Random Functions
por: Teney, Damien, et al.
Publicado: (2024)

Candidate Set Re-ranking for Composed Image Retrieval with Dual Multi-modal Encoder
por: Liu, Zheyuan, et al.
Publicado: (2023)

Efficient Learning With Sine-Activated Low-rank Matrices
por: Ji, Yiping, et al.
Publicado: (2024)

RandLoRA: Full-rank parameter-efficient fine-tuning of large models
por: Albert, Paul, et al.
Publicado: (2025)

Enhancing Transformers Through Conditioned Embedded Tokens
por: Saratchandran, Hemanth, et al.
Publicado: (2025)

Chem4DLLM: 4D Multimodal LLMs for Chemical Dynamics Understanding
por: Li, Xinyu, et al.
Publicado: (2026)

RanPAC: Random Projections and Pre-trained Models for Continual Learning
por: McDonnell, Mark D., et al.
Publicado: (2023)

Scalable Ensemble Diversification for OOD Generalization and Detection
por: Rubinstein, Alexander, et al.
Publicado: (2024)

On the Value of Cross-Modal Misalignment in Multimodal Representation Learning
por: Cai, Yichao, et al.
Publicado: (2025)