Beware of the Batch Size: Hyperparameter Bias in Evaluating LoRA
Fuente:
arXiv
Guardado en:
| Autores principales: | , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2026
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866913178531659776 |
|---|---|
| author | Lee, Sangyoon Lee, Jaeho |
| author_facet | Lee, Sangyoon Lee, Jaeho |
| contents | Low-rank adaptation (LoRA) is a standard approach for fine-tuning large language models, yet its many variants report conflicting empirical gains, often on the same benchmarks. We show that these contradictions arise from a single overlooked factor: the batch size. When properly tuned, vanilla LoRA often matches the performance of more complex variants. We further propose a proxy-based, cost-efficient strategy for batch size tuning, revealing the impact of rank, dataset size, and model capacity on the optimal batch size. Our findings elevate batch size from a minor implementation detail to a first-order design parameter, reconciling prior inconsistencies and enabling more reliable evaluations of LoRA variants. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2602_09492 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | Beware of the Batch Size: Hyperparameter Bias in Evaluating LoRA Lee, Sangyoon Lee, Jaeho Machine Learning Artificial Intelligence Low-rank adaptation (LoRA) is a standard approach for fine-tuning large language models, yet its many variants report conflicting empirical gains, often on the same benchmarks. We show that these contradictions arise from a single overlooked factor: the batch size. When properly tuned, vanilla LoRA often matches the performance of more complex variants. We further propose a proxy-based, cost-efficient strategy for batch size tuning, revealing the impact of rank, dataset size, and model capacity on the optimal batch size. Our findings elevate batch size from a minor implementation detail to a first-order design parameter, reconciling prior inconsistencies and enabling more reliable evaluations of LoRA variants. |
| title | Beware of the Batch Size: Hyperparameter Bias in Evaluating LoRA |
| topic | Machine Learning Artificial Intelligence |
| url | https://arxiv.org/abs/2602.09492 |