Cover Image

How does Multi-Task Training Affect Transformer In-Context Capabilities? Investigations with Function Classes

Fuente: arXiv

Saved in:

Bibliographic Details
Main Authors:	Bhasin, Harmon, Ossowski, Timothy, Zhong, Yiqiao, Hu, Junjie
Format:	Preprint
Published:	2024
Subjects:	Computation and Language Machine Learning
Online Access:	Acceder al recurso
Tags:	Add Tag No Tags, Be the first to tag this record!

Similar Items

Task Vector Geometry Underlies Dual Modes of Task Inference in Transformers
by: Yan, Hao, et al.
Published: (2026)

Investigating the Pre-Training Dynamics of In-Context Learning: Task Recognition vs. Task Learning
by: Wang, Xiaolei, et al.
Published: (2024)

Out-of-distribution generalization via composition: a lens through induction heads in Transformers
by: Song, Jiajun, et al.
Published: (2024)

How Numerical Precision Affects Arithmetical Reasoning Capabilities of LLMs
by: Feng, Guhao, et al.
Published: (2024)

How Does Code Pretraining Affect Language Model Task Performance?
by: Petty, Jackson, et al.
Published: (2024)

On Mesa-Optimization in Autoregressively Trained Transformers: Emergence and Capability
by: Zheng, Chenyu, et al.
Published: (2024)

Algorithmic Capabilities of Random Transformers
by: Zhong, Ziqian, et al.
Published: (2024)

Contextual Drag: How Errors in the Context Affect LLM Reasoning
by: Cheng, Yun, et al.
Published: (2026)

How to Train Long-Context Language Models (Effectively)
by: Gao, Tianyu, et al.
Published: (2024)

Memory Is All You Need: Testing How Model Memory Affects LLM Performance in Annotation Tasks
by: Timoneda, Joan C., et al.
Published: (2025)

An Analysis of Embedding Layers and Similarity Scores using Siamese Neural Networks
by: Bingi, Yash, et al.
Published: (2023)

HMS-BERT: Hybrid Multi-Task Self-Training for Multilingual and Multi-Label Cyberbullying Detection
by: Feng, Zixin, et al.
Published: (2026)

MultiGPrompt for Multi-Task Pre-Training and Prompting on Graphs
by: Yu, Xingtong, et al.
Published: (2023)

Learning Task Representations from In-Context Learning
by: Saglam, Baturay, et al.
Published: (2025)

Mapping Post-Training Forgetting in Language Models at Scale
by: Harmon, Jackson, et al.
Published: (2025)

Context-Alignment: Activating and Enhancing LLM Capabilities in Time Series
by: Hu, Yuxiao, et al.
Published: (2025)

CICLe: Conformal In-Context Learning for Largescale Multi-Class Food Risk Classification
by: Randl, Korbinian, et al.
Published: (2024)

Linear Transformers with Learnable Kernel Functions are Better In-Context Models
by: Aksenov, Yaroslav, et al.
Published: (2024)

Investigating Symbolic Capabilities of Large Language Models
by: Dave, Neisarg, et al.
Published: (2024)

How Does Quantization Affect Multilingual LLMs?
by: Marchisio, Kelly, et al.
Published: (2024)

Capability-Oriented Training Induced Alignment Risk
by: Zhou, Yujun, et al.
Published: (2026)

Balancing Synthetic Data and Replay for Enhancing Task-Specific Capabilities
by: Spiegelhalter, Urs, et al.
Published: (2025)

OLIVE: Object Level In-Context Visual Embeddings
by: Ossowski, Timothy, et al.
Published: (2024)

NeuroLoRA: Context-Aware Neuromodulation for Parameter-Efficient Multi-Task Adaptation
by: Yang, Yuxin, et al.
Published: (2026)

Understanding Contextual Recall in Transformers: How Finetuning Enables In-Context Reasoning over Pretraining Knowledge
by: Vasudeva, Bhavya, et al.
Published: (2026)

A Multi-Level Framework for Accelerating Training Transformer Models
by: Zou, Longwei, et al.
Published: (2024)

Interpreting Context Look-ups in Transformers: Investigating Attention-MLP Interactions
by: Neo, Clement, et al.
Published: (2024)

From Lists to Emojis: How Format Bias Affects Model Alignment
by: Zhang, Xuanchang, et al.
Published: (2024)

Transformers Learn Low Sensitivity Functions: Investigations and Implications
by: Vasudeva, Bhavya, et al.
Published: (2024)

Assessing LLM Text Detection in Educational Contexts: Does Human Contribution Affect Detection?
by: Gehring, Lukas, et al.
Published: (2025)

Collaborate, Deliberate, Evaluate: How LLM Alignment Affects Coordinated Multi-Agent Outcomes
by: Nath, Abhijnan, et al.
Published: (2025)

Core Context Aware Transformers for Long Context Language Modeling
by: Chen, Yaofo, et al.
Published: (2024)

In-Context Occam's Razor: How Transformers Prefer Simpler Hypotheses on the Fly
by: Deora, Puneesh, et al.
Published: (2025)

In-Context Learning with Transformers: Softmax Attention Adapts to Function Lipschitzness
by: Collins, Liam, et al.
Published: (2024)

Efficient Knowledge Probing of Large Language Models by Adapting Pre-trained Embeddings
by: Sharma, Kartik, et al.
Published: (2025)

How Transformers Learn Regular Language Recognition: A Theoretical Study on Training Dynamics and Implicit Bias
by: Huang, Ruiquan, et al.
Published: (2025)

Investigating Affect Mining Techniques for Annotation Sample Selection in the Creation of Finnish Affective Speech Corpus
by: Lahtinen, Kalle, et al.
Published: (2025)

Evolving LLMs' Self-Refinement Capability via Synergistic Training-Inference Optimization
by: Zeng, Yongcheng, et al.
Published: (2025)

The Geometric Anatomy of Capability Acquisition in Transformers
by: Billa, Jayadev
Published: (2026)

Frustratingly Easy Task-aware Pruning for Large Language Models
by: Tian, Yuanhe, et al.
Published: (2025)