Cover Image

Your Transformer is Secretly Linear

Fuente: arXiv

Saved in:

Bibliographic Details
Main Authors:	Razzhigaev, Anton, Mikhalchuk, Matvey, Goncharova, Elizaveta, Gerasimenko, Nikolai, Oseledets, Ivan, Dimitrov, Denis, Kuznetsov, Andrey
Format:	Preprint
Published:	2024
Subjects:	Machine Learning Artificial Intelligence Computation and Language
Online Access:	Acceder al recurso
Tags:	Add Tag No Tags, Be the first to tag this record!

Similar Items

The Shape of Learning: Anisotropy and Intrinsic Dimensions in Transformer-Based Models
by: Razzhigaev, Anton, et al.
Published: (2023)

LLM-Microscope: Uncovering the Hidden Role of Punctuation in Context Memory of Transformers
by: Razzhigaev, Anton, et al.
Published: (2025)

OmniFusion Technical Report
by: Goncharova, Elizaveta, et al.
Published: (2024)

Addressing Hallucinations in Language Models with Knowledge Graph Embeddings as an Additional Modality
by: Chekalina, Viktoriia, et al.
Published: (2024)

Back to Basics: Revisiting Exploration in Reinforcement Learning for LLM Reasoning via Generative Probabilities
by: Li, Pengyi, et al.
Published: (2026)

Logit-KL Flow Matching: Non-Autoregressive Text Generation via Sampling-Hybrid Inference
by: Sevriugov, Egor, et al.
Published: (2024)

Simple Vision-Language Math Reasoning via Rendered Text
by: Skripkin, Matvey, et al.
Published: (2025)

Inverting Black-Box Face Recognition Systems via Zero-Order Optimization in Eigenface Space
by: Razzhigaev, Anton, et al.
Published: (2025)

On the Spatial Structure of Mixture-of-Experts in Transformers
by: Bershatsky, Daniel, et al.
Published: (2025)

Confidence Is All You Need: Few-Shot RL Fine-Tuning of Language Models
by: Li, Pengyi, et al.
Published: (2025)

MindShift: Analyzing Language Models' Reactions to Psychological Prompts
by: Vasiliuk, Anton, et al.
Published: (2025)

Exploring the Hidden Capacity of LLMs for One-Step Text Generation
by: Mezentsev, Gleb, et al.
Published: (2025)

Universal Adversarial Attack on Aligned Multimodal LLMs
by: Rahmatullaev, Temurbek, et al.
Published: (2025)

CRAFT: Cultural Russian-Oriented Dataset Adaptation for Focused Text-to-Image Generation
by: Vasilev, Viacheslav, et al.
Published: (2025)

Real-World Transferable Adversarial Attack on Face-Recognition Systems
by: Kaznacheev, Andrey, et al.
Published: (2025)

Your Absorbing Discrete Diffusion Secretly Models the Bayesian Posterior
by: Doyle, Cooper
Published: (2025)

SONAR-LLM: Autoregressive Transformer that Thinks in Sentence Embeddings and Speaks in Tokens
by: Dragunov, Nikita, et al.
Published: (2025)

Direct Preference Optimization: Your Language Model is Secretly a Reward Model
by: Rafailov, Rafael, et al.
Published: (2023)

Check Your LLM's Secret Dictionary! Five Lines of Code Reveal What Your LLM Learned (Including What It Shouldn't Have)
by: Miyashita, Hisashi
Published: (2026)

LoTR: Low Tensor Rank Weight Adaptation
by: Bershatsky, Daniel, et al.
Published: (2024)

GigaEvo: An Open Source Optimization Framework Powered By LLMs And Evolution Algorithms
by: Khrulkov, Valentin, et al.
Published: (2025)

LightTransfer: Your Long-Context LLM is Secretly a Hybrid Model with Effortless Adaptation
by: Zhang, Xuan, et al.
Published: (2024)

KV Cache Offloading for Context-Intensive Tasks
by: Bocharnikov, Andrey, et al.
Published: (2026)

RusCode: Russian Cultural Code Benchmark for Text-to-Image Generation
by: Vasilev, Viacheslav, et al.
Published: (2025)

BgGPT 1.0: Extending English-centric LLMs to other languages
by: Alexandrov, Anton, et al.
Published: (2024)

Sentence-Anchored Gist Compression for Long-Context LLMs
by: Tarasov, Dmitrii, et al.
Published: (2025)

Subliminal Effects in Your Data: A General Mechanism via Log-Linearity
by: Aden-Ali, Ishaq, et al.
Published: (2026)

Share Your Attention: Transformer Weight Sharing via Matrix-based Dictionary Learning
by: Zhussip, Magauiya, et al.
Published: (2025)

ESQA: Event Sequences Question Answering
by: Abdullaeva, Irina, et al.
Published: (2024)

On the Robustness of Transformers against Context Hijacking for Linear Classification
by: Li, Tianle, et al.
Published: (2025)

Use Your INSTINCT: INSTruction optimization for LLMs usIng Neural bandits Coupled with Transformers
by: Lin, Xiaoqiang, et al.
Published: (2023)

Transformer Block Coupling and its Correlation with Generalization in LLMs
by: Aubry, Murdock, et al.
Published: (2024)

Cross-Architecture Transfer Learning for Linear-Cost Inference Transformers
by: Choi, Sehyun
Published: (2024)

Latent Reasoning in TRMs is Secretly a Policy Improvement Operator
by: Asadulaev, Arip, et al.
Published: (2025)

Censored LLMs as a Natural Testbed for Secret Knowledge Elicitation
by: Casademunt, Helena, et al.
Published: (2026)

On the Bias of Next-Token Predictors Toward Systematically Inefficient Reasoning: A Shortest-Path Case Study
by: Alberghi, Riccardo, et al.
Published: (2025)

Multi-Layer Transformers Gradient Can be Approximated in Almost Linear Time
by: Liang, Yingyu, et al.
Published: (2024)

mini-vec2vec: Scaling Universal Geometry Alignment with Linear Transformations
by: Dar, Guy
Published: (2025)

Transformers Learn to Achieve Second-Order Convergence Rates for In-Context Linear Regression
by: Fu, Deqing, et al.
Published: (2023)

ReplaceMe: Network Simplification via Depth Pruning and Transformer Block Linearization
by: Shopkhoev, Dmitriy, et al.
Published: (2025)