Fine-Grained Embedding Dimension Optimization During Training for Recommender Systems
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2024
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866910647191601152 |
|---|---|
| author | Luo, Qinyi Wang, Penghan Zhang, Wei Lai, Fan Mao, Jiachen Wei, Xiaohan Song, Jun Tsai, Wei-Yu Yang, Shuai Hu, Yuxi Qian, Xuehai |
| author_facet | Luo, Qinyi Wang, Penghan Zhang, Wei Lai, Fan Mao, Jiachen Wei, Xiaohan Song, Jun Tsai, Wei-Yu Yang, Shuai Hu, Yuxi Qian, Xuehai |
| contents | Huge embedding tables in modern deep learning recommender models (DLRM) require prohibitively large memory during training and inference. This paper proposes FIITED, a system to automatically reduce the memory footprint via FIne-grained In-Training Embedding Dimension pruning. By leveraging the key insight that embedding vectors are not equally important, FIITED adaptively adjusts the dimension of each individual embedding vector during model training, assigning larger dimensions to more important embeddings while adapting to dynamic changes in data. We prioritize embedding dimensions with higher frequencies and gradients as more important. To enable efficient pruning of embeddings and their dimensions during model training, we propose an embedding storage system based on virtually-hashed physically-indexed hash tables. Experiments on two industry models and months of realistic datasets show that FIITED can reduce DLRM embedding size by more than 65% while preserving model quality, outperforming state-of-the-art in-training embedding pruning methods. On public datasets, FIITED can reduce the size of embedding tables by 2.1x to 800x with negligible accuracy drop, while improving model throughput. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2401_04408 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Fine-Grained Embedding Dimension Optimization During Training for Recommender Systems Luo, Qinyi Wang, Penghan Zhang, Wei Lai, Fan Mao, Jiachen Wei, Xiaohan Song, Jun Tsai, Wei-Yu Yang, Shuai Hu, Yuxi Qian, Xuehai Information Retrieval Machine Learning I.2.6; H.3.3 Huge embedding tables in modern deep learning recommender models (DLRM) require prohibitively large memory during training and inference. This paper proposes FIITED, a system to automatically reduce the memory footprint via FIne-grained In-Training Embedding Dimension pruning. By leveraging the key insight that embedding vectors are not equally important, FIITED adaptively adjusts the dimension of each individual embedding vector during model training, assigning larger dimensions to more important embeddings while adapting to dynamic changes in data. We prioritize embedding dimensions with higher frequencies and gradients as more important. To enable efficient pruning of embeddings and their dimensions during model training, we propose an embedding storage system based on virtually-hashed physically-indexed hash tables. Experiments on two industry models and months of realistic datasets show that FIITED can reduce DLRM embedding size by more than 65% while preserving model quality, outperforming state-of-the-art in-training embedding pruning methods. On public datasets, FIITED can reduce the size of embedding tables by 2.1x to 800x with negligible accuracy drop, while improving model throughput. |
| title | Fine-Grained Embedding Dimension Optimization During Training for Recommender Systems |
| topic | Information Retrieval Machine Learning I.2.6; H.3.3 |
| url | https://arxiv.org/abs/2401.04408 |