Domain Adaptation of Foundation LLMs for e-Commerce
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866908377431408640 |
|---|---|
| author | Herold, Christian Kozielski, Michael Bazazo, Tala Petrushkov, Pavel Cieplicka, Patrycja Basaj, Dominika Versley, Yannick Hashemi, Seyyed Hadi Khadivi, Shahram |
| author_facet | Herold, Christian Kozielski, Michael Bazazo, Tala Petrushkov, Pavel Cieplicka, Patrycja Basaj, Dominika Versley, Yannick Hashemi, Seyyed Hadi Khadivi, Shahram |
| contents | We present the e-Llama models: 8 billion and 70 billion parameter large language models that are adapted towards the e-commerce domain. These models are meant as foundation models with deep knowledge about e-commerce, that form a base for instruction- and fine-tuning. The e-Llama models are obtained by continuously pretraining the Llama 3.1 base models on 1 trillion tokens of domain-specific data.
We discuss our approach and motivate our choice of hyperparameters with a series of ablation studies. To quantify how well the models have been adapted to the e-commerce domain, we define and implement a set of multilingual, e-commerce specific evaluation tasks.
We show that, when carefully choosing the training setup, the Llama 3.1 models can be adapted towards the new domain without sacrificing significant performance on general domain tasks. We also explore the possibility of merging the adapted model and the base model for a better control of the performance trade-off between domains. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2501_09706 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Domain Adaptation of Foundation LLMs for e-Commerce Herold, Christian Kozielski, Michael Bazazo, Tala Petrushkov, Pavel Cieplicka, Patrycja Basaj, Dominika Versley, Yannick Hashemi, Seyyed Hadi Khadivi, Shahram Computation and Language We present the e-Llama models: 8 billion and 70 billion parameter large language models that are adapted towards the e-commerce domain. These models are meant as foundation models with deep knowledge about e-commerce, that form a base for instruction- and fine-tuning. The e-Llama models are obtained by continuously pretraining the Llama 3.1 base models on 1 trillion tokens of domain-specific data. We discuss our approach and motivate our choice of hyperparameters with a series of ablation studies. To quantify how well the models have been adapted to the e-commerce domain, we define and implement a set of multilingual, e-commerce specific evaluation tasks. We show that, when carefully choosing the training setup, the Llama 3.1 models can be adapted towards the new domain without sacrificing significant performance on general domain tasks. We also explore the possibility of merging the adapted model and the base model for a better control of the performance trade-off between domains. |
| title | Domain Adaptation of Foundation LLMs for e-Commerce |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2501.09706 |