Efficient Many-Shot In-Context Learning with Dynamic Block-Sparse Attention
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2025
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866909541318262784 |
|---|---|
| author | Xiao, Emily Li, Chin-Jou Zhang, Yilin Neubig, Graham Bertsch, Amanda |
| author_facet | Xiao, Emily Li, Chin-Jou Zhang, Yilin Neubig, Graham Bertsch, Amanda |
| contents | Many-shot in-context learning has recently shown promise as an alternative to finetuning, with the major advantage that the same model can be served for multiple tasks. However, this shifts the computational burden from training-time to inference-time, making deployment of many-shot ICL challenging to justify in-practice. This cost is further increased if a custom demonstration set is retrieved for each inference example. We present Dynamic Block-Sparse Attention, a training-free framework for retrieval-based many-shot in-context learning. By combining carefully designed block-sparse attention and retrieval of cached groups of demonstrations, we achieve comparable per-example latency to finetuning while maintaining on average >95% of the best method's accuracy across strong ICL and finetuning baselines. We hope that this will further enable the deployment of many-shot ICL at scale. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2503_08640 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Efficient Many-Shot In-Context Learning with Dynamic Block-Sparse Attention Xiao, Emily Li, Chin-Jou Zhang, Yilin Neubig, Graham Bertsch, Amanda Computation and Language Many-shot in-context learning has recently shown promise as an alternative to finetuning, with the major advantage that the same model can be served for multiple tasks. However, this shifts the computational burden from training-time to inference-time, making deployment of many-shot ICL challenging to justify in-practice. This cost is further increased if a custom demonstration set is retrieved for each inference example. We present Dynamic Block-Sparse Attention, a training-free framework for retrieval-based many-shot in-context learning. By combining carefully designed block-sparse attention and retrieval of cached groups of demonstrations, we achieve comparable per-example latency to finetuning while maintaining on average >95% of the best method's accuracy across strong ICL and finetuning baselines. We hope that this will further enable the deployment of many-shot ICL at scale. |
| title | Efficient Many-Shot In-Context Learning with Dynamic Block-Sparse Attention |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2503.08640 |