Saved in:
| Main Authors: | , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2509.15561 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866916969149628416 |
|---|---|
| author | Naphade, Om Bansal, Saksham Pareek, Parikshit |
| author_facet | Naphade, Om Bansal, Saksham Pareek, Parikshit |
| contents | Hyper-parameter Tuning (HPT) is a necessary step in machine learning (ML) pipelines but becomes computationally expensive and opaque with larger models. Recently, Large Language Models (LLMs) have been explored for HPT, yet most rely on models exceeding 100 billion parameters. We propose an Expert Block Framework for HPT using Small LLMs. At its core is the Trajectory Context Summarizer (TCS), a deterministic block that transforms raw training trajectories into structured context, enabling small LLMs to analyze optimization progress with reliability comparable to larger models. Using two locally-run LLMs (phi4:reasoning14B and qwen2.5-coder:32B) and a 10-trial budget, our TCS-enabled HPT pipeline achieves average performance within ~0.9 percentage points of GPT-4 across six diverse tasks. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2509_15561 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Small LLMs with Expert Blocks Are Good Enough for Hyperparamter Tuning Naphade, Om Bansal, Saksham Pareek, Parikshit Machine Learning Computation and Language Hyper-parameter Tuning (HPT) is a necessary step in machine learning (ML) pipelines but becomes computationally expensive and opaque with larger models. Recently, Large Language Models (LLMs) have been explored for HPT, yet most rely on models exceeding 100 billion parameters. We propose an Expert Block Framework for HPT using Small LLMs. At its core is the Trajectory Context Summarizer (TCS), a deterministic block that transforms raw training trajectories into structured context, enabling small LLMs to analyze optimization progress with reliability comparable to larger models. Using two locally-run LLMs (phi4:reasoning14B and qwen2.5-coder:32B) and a 10-trial budget, our TCS-enabled HPT pipeline achieves average performance within ~0.9 percentage points of GPT-4 across six diverse tasks. |
| title | Small LLMs with Expert Blocks Are Good Enough for Hyperparamter Tuning |
| topic | Machine Learning Computation and Language |
| url | https://arxiv.org/abs/2509.15561 |