Image de couverture de livre

In-Context Learning with Transformers: Softmax Attention Adapts to Function Lipschitzness

Fuente: arXiv

Enregistré dans:

Détails bibliographiques
Auteurs principaux:	Collins, Liam, Parulekar, Advait, Mokhtari, Aryan, Sanghavi, Sujay, Shakkottai, Sanjay
Format:	Preprint
Publié:	2024
Sujets:	Machine Learning Artificial Intelligence Computation and Language
Accès en ligne:	Acceder al recurso
Tags:	Ajouter un tag Pas de tags, Soyez le premier à ajouter un tag!

Documents similaires

Entropy Aware Reward Guidance for Diffusion Language Model Alignment
par: Tejaswi, Atula, et autres
Publié: (2026)

Efficient Approximate Posterior Sampling with Annealed Langevin Monte Carlo
par: Parulekar, Advait, et autres
Publié: (2025)

Machine Unlearning under Overparameterization
par: Block, Jacob L., et autres
Publié: (2025)

HiSpec: Hierarchical Speculative Decoding for LLMs
par: Kumar, Avinash, et autres
Publié: (2025)

When Attention Collapses: How Degenerate Layers in LLMs Enable Smaller, Stronger Models
par: Sanyal, Sunny, et autres
Publié: (2024)

Forgetting Transformer: Softmax Attention with a Forget Gate
par: Lin, Zhixuan, et autres
Publié: (2025)

Scalable-Softmax Is Superior for Attention
par: Nakanishi, Ken M.
Publié: (2025)

To Softmax, or not to Softmax: that is the question when applying Active Learning for Transformer Models
par: Gonsior, Julius, et autres
Publié: (2022)

Provable Multi-Task Representation Learning by Two-Layer ReLU Neural Networks
par: Collins, Liam, et autres
Publié: (2023)

Sculpting Latent Spaces With MMD: Disentanglement With Programmable Priors
par: Fruytier, Quentin, et autres
Publié: (2025)

Learning Mixtures of Experts with EM: A Mirror Descent Perspective
par: Fruytier, Quentin, et autres
Publié: (2024)

Provable Meta-Learning with Low-Rank Adaptations
par: Block, Jacob L., et autres
Publié: (2024)

CARE-RFT: Confidence-Anchored Reinforcement Finetuning for Reliable Reasoning in Large Language Models
par: Li, Shuozhe, et autres
Publié: (2026)

LASER: An LLM-based ASR Scoring and Evaluation Rubric
par: Parulekar, Amruta, et autres
Publié: (2025)

Unmask It! AI-Generated Product Review Detection in Dravidian Languages
par: De, Somsubhra, et autres
Publié: (2025)

Enabling Approximate Joint Sampling in Diffusion LMs
par: Bansal, Parikshit, et autres
Publié: (2025)

The Information Geometry of Softmax: Probing and Steering
par: Park, Kiho, et autres
Publié: (2026)

Interpreting Context Look-ups in Transformers: Investigating Attention-MLP Interactions
par: Neo, Clement, et autres
Publié: (2024)

Which Attention Heads Matter for In-Context Learning?
par: Yin, Kayo, et autres
Publié: (2025)

Toward In-Context Teaching: Adapting Examples to Students' Misconceptions
par: Ross, Alexis, et autres
Publié: (2024)

SVFT: Parameter-Efficient Fine-Tuning with Singular Vectors
par: Lingam, Vijay, et autres
Publié: (2024)

OptRot: Mitigating Weight Outliers via Data-Free Rotations for Post-Training Quantization
par: Gadhikar, Advait, et autres
Publié: (2025)

PITA: Preference-Guided Inference-Time Alignment for LLM Post-Training
par: Bobbili, Sarat Chandra, et autres
Publié: (2025)

RARe: Retrieval Augmented Retrieval with In-Context Examples
par: Tejaswi, Atula, et autres
Publié: (2024)

Blocking Bandits
par: Basu, Soumya, et autres
Publié: (2019)

Adapting LLMs for Efficient Context Processing through Soft Prompt Compression
par: Wang, Cangqing, et autres
Publié: (2024)

Illuminate: A novel approach for depression detection with explainable analysis and proactive therapy using prompt engineering
par: Agrawal, Aryan
Publié: (2024)

Inference and Verbalization Functions During In-Context Learning
par: Tao, Junyi, et autres
Publié: (2024)

Do pretrained Transformers Learn In-Context by Gradient Descent?
par: Shen, Lingfeng, et autres
Publié: (2023)

CoCA: Fusing Position Embedding with Collinear Constrained Attention in Transformers for Long Context Window Extending
par: Zhu, Shiyi, et autres
Publié: (2023)

Geometric Median (GM) Matching for Robust Data Pruning
par: Acharya, Anish, et autres
Publié: (2024)

TransformLLM: Adapting Large Language Models via LLM-Transformed Reading Comprehension Text
par: Arbel, Iftach, et autres
Publié: (2024)

AdaptAgent: Adapting Multimodal Web Agents with Few-Shot Learning from Human Demonstrations
par: Verma, Gaurav, et autres
Publié: (2024)

Decomposing Attention To Find Context-Sensitive Neurons
par: Gibson, Alex
Publié: (2025)

Selective Attention Improves Transformer
par: Leviathan, Yaniv, et autres
Publié: (2024)

Test-Time Speculation
par: Kumar, Avinash, et autres
Publié: (2026)

AMPS: ASR with Multimodal Paraphrase Supervision
par: Gupta, Abhishek, et autres
Publié: (2024)

Parameter-efficient Adaptation of Multilingual Multimodal Models for Low-resource ASR
par: Gupta, Abhishek, et autres
Publié: (2024)

Attention Mechanisms Don't Learn Additive Models: Rethinking Feature Importance for Transformers
par: Leemann, Tobias, et autres
Publié: (2024)

SEAL: Scaling to Emphasize Attention for Long-Context Retrieval
par: Lee, Changhun, et autres
Publié: (2025)