Saved in:
| Main Authors: | , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2026
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2605.16686 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866914571586895872 |
|---|---|
| author | Maksimov, Roman Aletov, Vladimir Bylinkin, Dmitry Medyakov, Daniil Solodkin, Vladimir Beznosikov, Aleksandr |
| author_facet | Maksimov, Roman Aletov, Vladimir Bylinkin, Dmitry Medyakov, Daniil Solodkin, Vladimir Beznosikov, Aleksandr |
| contents | Knowledge editing (KE) provides a lightweight alternative to repeated fine-tuning of LLMs. However, most existing KE methods target dense feed-forward layers, while modern LLMs increasingly adopt Mixture-of-Experts (MoE) architectures for their superior memory footprint and inference efficiency. This mismatch leaves a growing class of production models without principled editing tools. We propose a MEMIT-like framework for knowledge editing in MoE-based LLMs. Our method exploits the tensor structure of MoE layers to formulate the editing objective faithfully at the per expert level, and applies the Woodbury matrix identity to avoid materializing or inverting the full stacked matrix of expert weights. The resulting update reduces to inversions of fixed low-rank matrices and requires no additional backward passes. Empirically, our approach matches the editing quality of strong baselines on the main KE metrics while accelerating the editing procedure by up to 6x, owing to the batched MEMIT-style formulation and the low-dimensional inversions enabled by the Woodbury identity. These results show that closed-form, parameter-modifying KE can be extended efficiently beyond dense layers, opening a path toward scalable knowledge editing in modern sparse LLM architectures. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2605_16686 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | Scalable Knowledge Editing for Mixture-of-Experts LLMs via Tensor-Structured Updates Maksimov, Roman Aletov, Vladimir Bylinkin, Dmitry Medyakov, Daniil Solodkin, Vladimir Beznosikov, Aleksandr Machine Learning Knowledge editing (KE) provides a lightweight alternative to repeated fine-tuning of LLMs. However, most existing KE methods target dense feed-forward layers, while modern LLMs increasingly adopt Mixture-of-Experts (MoE) architectures for their superior memory footprint and inference efficiency. This mismatch leaves a growing class of production models without principled editing tools. We propose a MEMIT-like framework for knowledge editing in MoE-based LLMs. Our method exploits the tensor structure of MoE layers to formulate the editing objective faithfully at the per expert level, and applies the Woodbury matrix identity to avoid materializing or inverting the full stacked matrix of expert weights. The resulting update reduces to inversions of fixed low-rank matrices and requires no additional backward passes. Empirically, our approach matches the editing quality of strong baselines on the main KE metrics while accelerating the editing procedure by up to 6x, owing to the batched MEMIT-style formulation and the low-dimensional inversions enabled by the Woodbury identity. These results show that closed-form, parameter-modifying KE can be extended efficiently beyond dense layers, opening a path toward scalable knowledge editing in modern sparse LLM architectures. |
| title | Scalable Knowledge Editing for Mixture-of-Experts LLMs via Tensor-Structured Updates |
| topic | Machine Learning |
| url | https://arxiv.org/abs/2605.16686 |