Saved in:
| Main Authors: | , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2512.00207 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866911294083301376 |
|---|---|
| author | Dugan, Owen Garcia, Roberto Junkins, Ronny Liu, Jerry Zinsley, Dylan Eyuboglu, Sabri Rudra, Atri Ré, Chris |
| author_facet | Dugan, Owen Garcia, Roberto Junkins, Ronny Liu, Jerry Zinsley, Dylan Eyuboglu, Sabri Rudra, Atri Ré, Chris |
| contents | The success of large language models (LLMs) can be attributed in part to their ability to efficiently store factual knowledge as key-value mappings within their MLP parameters. Recent work has proposed explicit weight constructions to build such fact-storing MLPs, providing an improved understanding of LLM fact storage mechanisms. In this paper, we introduce an MLP construction framework that improves over previous constructions in three areas: it 1) works for all but a measure-zero set of feasible input-output pairs, 2) achieves asymptotically optimal parameter efficiency matching information-theoretic bounds for some embeddings, and 3) maintains usability within Transformers for factual recall. Through our improvements, we 1) discover a metric on value embeddings that characterizes facts-per-parameter scaling for both constructed and gradient-descent-trained MLPs, 2) identify a simple encoder-decoder mechanism that empirically matches gradient-descent MLP facts-per-parameter asymptotics across all the inputs and outputs we test, and 3) uncover a fundamental tradeoff between an MLP's fact-storage capacity and its usability within Transformers. Finally, we demonstrate a proof-of-concept application of fact-storing MLPs: modular fact editing on one-layer Transformers by \textit{replacing entire MLPs at once}. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2512_00207 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Constructing Efficient Fact-Storing MLPs for Transformers Dugan, Owen Garcia, Roberto Junkins, Ronny Liu, Jerry Zinsley, Dylan Eyuboglu, Sabri Rudra, Atri Ré, Chris Machine Learning Artificial Intelligence The success of large language models (LLMs) can be attributed in part to their ability to efficiently store factual knowledge as key-value mappings within their MLP parameters. Recent work has proposed explicit weight constructions to build such fact-storing MLPs, providing an improved understanding of LLM fact storage mechanisms. In this paper, we introduce an MLP construction framework that improves over previous constructions in three areas: it 1) works for all but a measure-zero set of feasible input-output pairs, 2) achieves asymptotically optimal parameter efficiency matching information-theoretic bounds for some embeddings, and 3) maintains usability within Transformers for factual recall. Through our improvements, we 1) discover a metric on value embeddings that characterizes facts-per-parameter scaling for both constructed and gradient-descent-trained MLPs, 2) identify a simple encoder-decoder mechanism that empirically matches gradient-descent MLP facts-per-parameter asymptotics across all the inputs and outputs we test, and 3) uncover a fundamental tradeoff between an MLP's fact-storage capacity and its usability within Transformers. Finally, we demonstrate a proof-of-concept application of fact-storing MLPs: modular fact editing on one-layer Transformers by \textit{replacing entire MLPs at once}. |
| title | Constructing Efficient Fact-Storing MLPs for Transformers |
| topic | Machine Learning Artificial Intelligence |
| url | https://arxiv.org/abs/2512.00207 |