Prototype Transformer: Towards Language Model Architectures Interpretable by Design
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2026
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866911740580593664 |
|---|---|
| author | Yordanov, Yordan Forasassi, Matteo Menzat, Bayar Wang, Ruizhi Qi, Chang Kaltenberger, Markus M'Charrak, Amine Salvatori, Tommaso Lukasiewicz, Thomas |
| author_facet | Yordanov, Yordan Forasassi, Matteo Menzat, Bayar Wang, Ruizhi Qi, Chang Kaltenberger, Markus M'Charrak, Amine Salvatori, Tommaso Lukasiewicz, Thomas |
| contents | While state-of-the-art language models (LMs) surpass most humans in certain domains, their reasoning remains largely opaque, reducing trust and increasing the risk of deception and hallucination. We introduce the Prototype Transformer (ProtoT), an autoregressive LM architecture that replaces the quadratic-cost self-attention module of the Transformer with a linear-cost module based on prototypes, which are learned parameter vectors. In ProtoT, prototypes create communication channels that aggregate contextual information at different time scales. We show that this structure leads prototypes to automatically capture nameable concepts, such as "woman", during training, offering a path toward interpreting model reasoning and making targeted edits to model behavior. Compared with baselines, ProtoT scales well with model and data size, is robust to input perturbations, and performs well on text generation and downstream tasks, including GLUE. These results suggest that ProtoT is a promising step toward autoregressive language models that are more interpretable by design. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2602_11852 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | Prototype Transformer: Towards Language Model Architectures Interpretable by Design Yordanov, Yordan Forasassi, Matteo Menzat, Bayar Wang, Ruizhi Qi, Chang Kaltenberger, Markus M'Charrak, Amine Salvatori, Tommaso Lukasiewicz, Thomas Artificial Intelligence Computation and Language Machine Learning I.2.6; I.2.7 While state-of-the-art language models (LMs) surpass most humans in certain domains, their reasoning remains largely opaque, reducing trust and increasing the risk of deception and hallucination. We introduce the Prototype Transformer (ProtoT), an autoregressive LM architecture that replaces the quadratic-cost self-attention module of the Transformer with a linear-cost module based on prototypes, which are learned parameter vectors. In ProtoT, prototypes create communication channels that aggregate contextual information at different time scales. We show that this structure leads prototypes to automatically capture nameable concepts, such as "woman", during training, offering a path toward interpreting model reasoning and making targeted edits to model behavior. Compared with baselines, ProtoT scales well with model and data size, is robust to input perturbations, and performs well on text generation and downstream tasks, including GLUE. These results suggest that ProtoT is a promising step toward autoregressive language models that are more interpretable by design. |
| title | Prototype Transformer: Towards Language Model Architectures Interpretable by Design |
| topic | Artificial Intelligence Computation and Language Machine Learning I.2.6; I.2.7 |
| url | https://arxiv.org/abs/2602.11852 |