Adaptive Low-Precision Training for Embeddings in Click-Through Rate Prediction
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2022
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866911977957228544 |
|---|---|
| author | Li, Shiwei Guo, Huifeng Hou, Lu Zhang, Wei Tang, Xing Tang, Ruiming Zhang, Rui Li, Ruixuan |
| author_facet | Li, Shiwei Guo, Huifeng Hou, Lu Zhang, Wei Tang, Xing Tang, Ruiming Zhang, Rui Li, Ruixuan |
| contents | Embedding tables are usually huge in click-through rate (CTR) prediction models. To train and deploy the CTR models efficiently and economically, it is necessary to compress their embedding tables at the training stage. To this end, we formulate a novel quantization training paradigm to compress the embeddings from the training stage, termed low-precision training (LPT). Also, we provide theoretical analysis on its convergence. The results show that stochastic weight quantization has a faster convergence rate and a smaller convergence error than deterministic weight quantization in LPT. Further, to reduce the accuracy degradation, we propose adaptive low-precision training (ALPT) that learns the step size (i.e., the quantization resolution) through gradient descent. Experiments on two real-world datasets confirm our analysis and show that ALPT can significantly improve the prediction accuracy, especially at extremely low bit widths. For the first time in CTR models, we successfully train 8-bit embeddings without sacrificing prediction accuracy. The code of ALPT is publicly available. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2212_05735 |
| institution | arXiv |
| publishDate | 2022 |
| record_format | arxiv |
| spellingShingle | Adaptive Low-Precision Training for Embeddings in Click-Through Rate Prediction Li, Shiwei Guo, Huifeng Hou, Lu Zhang, Wei Tang, Xing Tang, Ruiming Zhang, Rui Li, Ruixuan Machine Learning Information Retrieval Embedding tables are usually huge in click-through rate (CTR) prediction models. To train and deploy the CTR models efficiently and economically, it is necessary to compress their embedding tables at the training stage. To this end, we formulate a novel quantization training paradigm to compress the embeddings from the training stage, termed low-precision training (LPT). Also, we provide theoretical analysis on its convergence. The results show that stochastic weight quantization has a faster convergence rate and a smaller convergence error than deterministic weight quantization in LPT. Further, to reduce the accuracy degradation, we propose adaptive low-precision training (ALPT) that learns the step size (i.e., the quantization resolution) through gradient descent. Experiments on two real-world datasets confirm our analysis and show that ALPT can significantly improve the prediction accuracy, especially at extremely low bit widths. For the first time in CTR models, we successfully train 8-bit embeddings without sacrificing prediction accuracy. The code of ALPT is publicly available. |
| title | Adaptive Low-Precision Training for Embeddings in Click-Through Rate Prediction |
| topic | Machine Learning Information Retrieval |
| url | https://arxiv.org/abs/2212.05735 |