Saved in:
| Main Authors: | , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2026
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2605.25344 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866913161167241216 |
|---|---|
| author | Lu, Ying Zhou, Peng-Fei Fang, Qi-Xuan Zhang, Pan Ran, Shi-Ju Su, Gang |
| author_facet | Lu, Ying Zhou, Peng-Fei Fang, Qi-Xuan Zhang, Pan Ran, Shi-Ju Su, Gang |
| contents | Large language models (LLMs) are dominated by dense linear transformations, whose storage, memory and computational overheads hinder efficient adaptation and deployment while masking the functional impacts of structural simplification. Here we present Tensor Mixture (MixT), a general tensor-structured compression scheme that replaces targeted dense linear layers with natively executable mixtures of tensor operators. Operating directly on generic linear projections instead of model-specific components, MixT is potentially applicable across Transformer-based LLMs and other dense neural mappings. We evaluate MixT on Qwen3-8B and LLaMA2-7B under a unified recovery protocol, identifying a broad compressible regime in which MMLU accuracy is largely preserved before an abrupt transition at model-specific boundaries. This transition coincides with coordinated shifts in output entropy, prediction entropy and inter-layer geometry. At the LLaMA2-7B transition boundary, MixT reduces full-model parameters by 47.5\%, inference FLOPs by 37.1\%, training FLOPs by 52.1\% and peak inference memory by 60.4\%, demonstrating its practical potential for lower-cost LLM compression. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2605_25344 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | A general tensor-structured compression scheme for efficient large language models Lu, Ying Zhou, Peng-Fei Fang, Qi-Xuan Zhang, Pan Ran, Shi-Ju Su, Gang Computation and Language Artificial Intelligence Machine Learning Quantum Physics Large language models (LLMs) are dominated by dense linear transformations, whose storage, memory and computational overheads hinder efficient adaptation and deployment while masking the functional impacts of structural simplification. Here we present Tensor Mixture (MixT), a general tensor-structured compression scheme that replaces targeted dense linear layers with natively executable mixtures of tensor operators. Operating directly on generic linear projections instead of model-specific components, MixT is potentially applicable across Transformer-based LLMs and other dense neural mappings. We evaluate MixT on Qwen3-8B and LLaMA2-7B under a unified recovery protocol, identifying a broad compressible regime in which MMLU accuracy is largely preserved before an abrupt transition at model-specific boundaries. This transition coincides with coordinated shifts in output entropy, prediction entropy and inter-layer geometry. At the LLaMA2-7B transition boundary, MixT reduces full-model parameters by 47.5\%, inference FLOPs by 37.1\%, training FLOPs by 52.1\% and peak inference memory by 60.4\%, demonstrating its practical potential for lower-cost LLM compression. |
| title | A general tensor-structured compression scheme for efficient large language models |
| topic | Computation and Language Artificial Intelligence Machine Learning Quantum Physics |
| url | https://arxiv.org/abs/2605.25344 |