Beyond Matryoshka: Revisiting Sparse Coding for Adaptive Representation
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866916744933670912 |
|---|---|
| author | Wen, Tiansheng Wang, Yifei Zeng, Zequn Peng, Zhong Su, Yudi Liu, Xinyang Chen, Bo Liu, Hongwei Jegelka, Stefanie You, Chenyu |
| author_facet | Wen, Tiansheng Wang, Yifei Zeng, Zequn Peng, Zhong Su, Yudi Liu, Xinyang Chen, Bo Liu, Hongwei Jegelka, Stefanie You, Chenyu |
| contents | Many large-scale systems rely on high-quality deep representations (embeddings) to facilitate tasks like retrieval, search, and generative modeling. Matryoshka Representation Learning (MRL) recently emerged as a solution for adaptive embedding lengths, but it requires full model retraining and suffers from noticeable performance degradations at short lengths. In this paper, we show that sparse coding offers a compelling alternative for achieving adaptive representation with minimal overhead and higher fidelity. We propose Contrastive Sparse Representation (CSR), a method that sparsifies pre-trained embeddings into a high-dimensional but selectively activated feature space. By leveraging lightweight autoencoding and task-aware contrastive objectives, CSR preserves semantic quality while allowing flexible, cost-effective inference at different sparsity levels. Extensive experiments on image, text, and multimodal benchmarks demonstrate that CSR consistently outperforms MRL in terms of both accuracy and retrieval speed-often by large margins-while also cutting training time to a fraction of that required by MRL. Our results establish sparse coding as a powerful paradigm for adaptive representation learning in real-world applications where efficiency and fidelity are both paramount. Code is available at https://github.com/neilwen987/CSR_Adaptive_Rep |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2503_01776 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Beyond Matryoshka: Revisiting Sparse Coding for Adaptive Representation Wen, Tiansheng Wang, Yifei Zeng, Zequn Peng, Zhong Su, Yudi Liu, Xinyang Chen, Bo Liu, Hongwei Jegelka, Stefanie You, Chenyu Machine Learning Artificial Intelligence Computer Vision and Pattern Recognition Information Retrieval Many large-scale systems rely on high-quality deep representations (embeddings) to facilitate tasks like retrieval, search, and generative modeling. Matryoshka Representation Learning (MRL) recently emerged as a solution for adaptive embedding lengths, but it requires full model retraining and suffers from noticeable performance degradations at short lengths. In this paper, we show that sparse coding offers a compelling alternative for achieving adaptive representation with minimal overhead and higher fidelity. We propose Contrastive Sparse Representation (CSR), a method that sparsifies pre-trained embeddings into a high-dimensional but selectively activated feature space. By leveraging lightweight autoencoding and task-aware contrastive objectives, CSR preserves semantic quality while allowing flexible, cost-effective inference at different sparsity levels. Extensive experiments on image, text, and multimodal benchmarks demonstrate that CSR consistently outperforms MRL in terms of both accuracy and retrieval speed-often by large margins-while also cutting training time to a fraction of that required by MRL. Our results establish sparse coding as a powerful paradigm for adaptive representation learning in real-world applications where efficiency and fidelity are both paramount. Code is available at https://github.com/neilwen987/CSR_Adaptive_Rep |
| title | Beyond Matryoshka: Revisiting Sparse Coding for Adaptive Representation |
| topic | Machine Learning Artificial Intelligence Computer Vision and Pattern Recognition Information Retrieval |
| url | https://arxiv.org/abs/2503.01776 |