CUDAHercules: Benchmarking Hardware-Aware Expert-level CUDA Optimization for LLMs
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2026
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866917474930262016 |
|---|---|
| author | Li, Shiyang Zhang, Zijian Sun, Guangyan Luo, Yuebo Chen, Winson Wang, Yanzhi Hong, Mingyi Ding, Caiwen |
| author_facet | Li, Shiyang Zhang, Zijian Sun, Guangyan Luo, Yuebo Chen, Winson Wang, Yanzhi Hong, Mingyi Ding, Caiwen |
| contents | Large language models show promise for automated CUDA programming, however even the strongest coding models (e.g., Claude-Opus-4.6) may still fall short of expert-level, architecture-aware optimization. We introduce CUDAHercules, a benchmark that evaluates generated CUDA against end-to-end human-expert SOTA systems. It spans single kernels, module-level operators, full applications, and unsolved challenge tasks across Ampere, Hopper, and Blackwell GPUs, with end-to-end tasks gated by domain-specific semantic validators. Evaluating models such as Claude-Opus-4.6 and GPT-5.4 shows a large gap between runnable CUDA and expert CUDA engineering: models often compile and pass tests, but rarely recover the optimization strategies needed to match expert performance. Application semantics further reduce success, and iterative or tool-augmented feedback can improve correctness while drifting toward slow fallback implementations. These results show that automated CUDA programming remains far from fully solved and requires stronger hardware reasoning, better tool use, and training objectives that connect code understanding to hardware architecture-grounded intelligence. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2605_08467 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | CUDAHercules: Benchmarking Hardware-Aware Expert-level CUDA Optimization for LLMs Li, Shiyang Zhang, Zijian Sun, Guangyan Luo, Yuebo Chen, Winson Wang, Yanzhi Hong, Mingyi Ding, Caiwen Machine Learning Large language models show promise for automated CUDA programming, however even the strongest coding models (e.g., Claude-Opus-4.6) may still fall short of expert-level, architecture-aware optimization. We introduce CUDAHercules, a benchmark that evaluates generated CUDA against end-to-end human-expert SOTA systems. It spans single kernels, module-level operators, full applications, and unsolved challenge tasks across Ampere, Hopper, and Blackwell GPUs, with end-to-end tasks gated by domain-specific semantic validators. Evaluating models such as Claude-Opus-4.6 and GPT-5.4 shows a large gap between runnable CUDA and expert CUDA engineering: models often compile and pass tests, but rarely recover the optimization strategies needed to match expert performance. Application semantics further reduce success, and iterative or tool-augmented feedback can improve correctness while drifting toward slow fallback implementations. These results show that automated CUDA programming remains far from fully solved and requires stronger hardware reasoning, better tool use, and training objectives that connect code understanding to hardware architecture-grounded intelligence. |
| title | CUDAHercules: Benchmarking Hardware-Aware Expert-level CUDA Optimization for LLMs |
| topic | Machine Learning |
| url | https://arxiv.org/abs/2605.08467 |