Enregistré dans:
| Auteurs principaux: | , , , , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2024
|
| Sujets: | |
| Accès en ligne: | https://arxiv.org/abs/2407.18990 |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866914903368925184 |
|---|---|
| author | Halfon, Alon Gretz, Shai Arviv, Ofir Spector, Artem Toledo-Ronen, Orith Katz, Yoav Ein-Dor, Liat Shmueli-Scheuer, Michal Slonim, Noam |
| author_facet | Halfon, Alon Gretz, Shai Arviv, Ofir Spector, Artem Toledo-Ronen, Orith Katz, Yoav Ein-Dor, Liat Shmueli-Scheuer, Michal Slonim, Noam |
| contents | Fine-tuning Large Language Models (LLMs) is an effective method to enhance their performance on downstream tasks. However, choosing the appropriate setting of tuning hyperparameters (HPs) is a labor-intensive and computationally expensive process. Here, we provide recommended HP configurations for practical use-cases that represent a better starting point for practitioners, when considering two SOTA LLMs and two commonly used tuning methods. We describe Coverage-based Search (CBS), a process for ranking HP configurations based on an offline extensive grid search, such that the top ranked configurations collectively provide a practical robust recommendation for a wide range of datasets and domains. We focus our experiments on Llama-3-8B and Mistral-7B, as well as full fine-tuning and LoRa, conducting a total of > 10,000 tuning experiments. Our results suggest that, in general, Llama-3-8B and LoRA should be preferred, when possible. Moreover, we show that for both models and tuning methods, exploring only a few HP configurations, as recommended by our analysis, can provide excellent results in practice, making this work a valuable resource for practitioners. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2407_18990 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Stay Tuned: An Empirical Study of the Impact of Hyperparameters on LLM Tuning in Real-World Applications Halfon, Alon Gretz, Shai Arviv, Ofir Spector, Artem Toledo-Ronen, Orith Katz, Yoav Ein-Dor, Liat Shmueli-Scheuer, Michal Slonim, Noam Machine Learning Artificial Intelligence Computation and Language Fine-tuning Large Language Models (LLMs) is an effective method to enhance their performance on downstream tasks. However, choosing the appropriate setting of tuning hyperparameters (HPs) is a labor-intensive and computationally expensive process. Here, we provide recommended HP configurations for practical use-cases that represent a better starting point for practitioners, when considering two SOTA LLMs and two commonly used tuning methods. We describe Coverage-based Search (CBS), a process for ranking HP configurations based on an offline extensive grid search, such that the top ranked configurations collectively provide a practical robust recommendation for a wide range of datasets and domains. We focus our experiments on Llama-3-8B and Mistral-7B, as well as full fine-tuning and LoRa, conducting a total of > 10,000 tuning experiments. Our results suggest that, in general, Llama-3-8B and LoRA should be preferred, when possible. Moreover, we show that for both models and tuning methods, exploring only a few HP configurations, as recommended by our analysis, can provide excellent results in practice, making this work a valuable resource for practitioners. |
| title | Stay Tuned: An Empirical Study of the Impact of Hyperparameters on LLM Tuning in Real-World Applications |
| topic | Machine Learning Artificial Intelligence Computation and Language |
| url | https://arxiv.org/abs/2407.18990 |