Imagen de Portada

How Many Pretraining Tasks Are Needed for In-Context Learning of Linear Regression?

Fuente: arXiv

Guardado en:

Detalles Bibliográficos
Autores principales:	Wu, Jingfeng, Zou, Difan, Chen, Zixiang, Braverman, Vladimir, Gu, Quanquan, Bartlett, Peter L.
Formato:	Preprint
Publicado:	2023
Materias:	Machine Learning
Acceso en línea:	Acceder al recurso
Etiquetas:	Agregar Etiqueta Sin Etiquetas, Sea el primero en etiquetar este registro!

Ejemplares similares

How Transformers Utilize Multi-Head Attention in In-Context Learning? A Case Study on Sparse Linear Regression
por: Chen, Xingwu, et al.
Publicado: (2024)

Improved Scaling Laws in Linear Regression via Data Reuse
por: Lin, Licong, et al.
Publicado: (2025)

In-Context Learning of a Linear Transformer Block: Benefits of the MLP Component and One-Step GD Initialization
por: Zhang, Ruiqi, et al.
Publicado: (2024)

Fixed Design Analysis of Regularization-Based Continual Learning
por: Li, Haoran, et al.
Publicado: (2023)

Towards Theoretical Understanding of Transformer Test-Time Computing: Investigation on In-Context Linear Regression
por: Chen, Xingwu, et al.
Publicado: (2025)

Large Stepsizes Accelerate Gradient Descent for Regularized Logistic Regression
por: Wu, Jingfeng, et al.
Publicado: (2025)

Memory-Statistics Tradeoff in Continual Learning with Structural Regularization
por: Li, Haoran, et al.
Publicado: (2025)

Risk Comparisons in Linear Regression: Implicit Regularization Dominates Explicit Regularization
por: Wu, Jingfeng, et al.
Publicado: (2025)

Learning under Quantization for High-Dimensional Linear Regression
por: Zhang, Dechen, et al.
Publicado: (2025)

Benefits of Early Stopping in Gradient Descent for Overparameterized Logistic Regression
por: Wu, Jingfeng, et al.
Publicado: (2025)

Minimax Optimal Convergence of Gradient Descent in Logistic Regression via Large and Adaptive Stepsizes
por: Zhang, Ruiqi, et al.
Publicado: (2025)

Scaling Laws in Linear Regression: Compute, Parameters, and Data
por: Lin, Licong, et al.
Publicado: (2024)

What Can Transformer Learn with Varying Depth? Case Studies on Sequence Learning Tasks
por: Chen, Xingwu, et al.
Publicado: (2024)

Convergence of Score-Based Discrete Diffusion Models: A Discrete-Time Analysis
por: Zhang, Zikun, et al.
Publicado: (2024)

Understanding SGD with Exponential Moving Average: A Case Study in Linear Regression
por: Li, Xuheng, et al.
Publicado: (2025)

Understanding Transferable Representation Learning and Zero-shot Transfer in CLIP
por: Chen, Zixiang, et al.
Publicado: (2023)

Context-Scaling versus Task-Scaling in In-Context Learning
por: Abedsoltan, Amirhesam, et al.
Publicado: (2024)

Scaling Laws for Precision in High-Dimensional Linear Regression
por: Zhang, Dechen, et al.
Publicado: (2026)

On the Collapse Errors Induced by the Deterministic Sampler for Diffusion Models
por: Zhang, Yi, et al.
Publicado: (2025)

Global Convergence and Rich Feature Learning in $L$-Layer Infinite-Width Neural Networks under $μ$P Parametrization
por: Chen, Zixiang, et al.
Publicado: (2025)

On the Interpolation Error of Nonlinear Attention versus Linear Regression
por: Liao, Zhenyu, et al.
Publicado: (2025)

On the Robustness of Transformers against Context Hijacking for Linear Classification
por: Li, Tianle, et al.
Publicado: (2025)

Large Stepsize Gradient Descent for Logistic Loss: Non-Monotonicity of the Loss Improves Optimization Efficiency
por: Wu, Jingfeng, et al.
Publicado: (2024)

Rephrase and Respond: Let Large Language Models Ask Better Questions for Themselves
por: Deng, Yihe, et al.
Publicado: (2023)

Matching the Statistical Query Lower Bound for $k$-Sparse Parity Problems with Sign Stochastic Gradient Descent
por: Kou, Yiwen, et al.
Publicado: (2024)

Diffusion Language Models Can Perform Many Tasks with Scaling and Instruction-Finetuning
por: Ye, Jiasheng, et al.
Publicado: (2023)

Learning Linear Regression with Low-Rank Tasks in-Context
por: Takanami, Kaito, et al.
Publicado: (2025)

On the Limitation and Experience Replay for GNNs in Continual Learning
por: Su, Junwei, et al.
Publicado: (2023)

Nearly Minimax Optimal Regret for Learning Linear Mixture Stochastic Shortest Path
por: Di, Qiwei, et al.
Publicado: (2024)

A Human-Like Reasoning Framework for Multi-Phases Planning Task with Large Language Models
por: Xie, Chengxing, et al.
Publicado: (2024)

Self-Play Fine-Tuning of Diffusion Models for Text-to-Image Generation
por: Yuan, Huizhuo, et al.
Publicado: (2024)

IM-Context: In-Context Learning for Imbalanced Regression Tasks
por: Nejjar, Ismail, et al.
Publicado: (2024)

How Does Critical Batch Size Scale in Pre-training?
por: Zhang, Hanlin, et al.
Publicado: (2024)

Why and How Auxiliary Tasks Improve JEPA Representations
por: Yu, Jiacan, et al.
Publicado: (2025)

MLPs Learn In-Context on Regression and Classification Tasks
por: Tong, William L., et al.
Publicado: (2024)

Self-Play Fine-Tuning Converts Weak Language Models to Strong Language Models
por: Chen, Zixiang, et al.
Publicado: (2024)

Learning with Incomplete Context: Linear Contextual Bandits with Pretrained Imputation
por: Yan, Hao, et al.
Publicado: (2025)

Pretrain-Test Task Alignment Governs Generalization in In-Context Learning
por: Letey, Mary I., et al.
Publicado: (2025)

Kernel Regression in Structured Non-IID Settings: Theory and Implications for Denoising Score Learning
por: Zhang, Dechen, et al.
Publicado: (2025)

Optimal Horizon-Free Reward-Free Exploration for Linear Mixture MDPs
por: Zhang, Junkai, et al.
Publicado: (2023)