Resource-efficient Inference with Foundation Model Programs
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866912529289052160 |
|---|---|
| author | Nie, Lunyiu Ding, Zhimin Yu, Kevin Cheung, Marco Jermaine, Chris Chaudhuri, Swarat |
| author_facet | Nie, Lunyiu Ding, Zhimin Yu, Kevin Cheung, Marco Jermaine, Chris Chaudhuri, Swarat |
| contents | The inference-time resource costs of large language and vision models present a growing challenge in production deployments. We propose the use of foundation model programs, i.e., programs that can invoke foundation models with varying resource costs and performance, as an approach to this problem. Specifically, we present a method that translates a task into a program, then learns a policy for resource allocation that, on each input, selects foundation model "backends" for each program module. The policy uses smaller, cheaper backends to handle simpler subtasks, while allowing more complex subtasks to leverage larger, more capable models. We evaluate the method on two new "streaming" visual question-answering tasks in which a system answers a question on a sequence of inputs, receiving ground-truth feedback after each answer. Compared to monolithic multi-modal models, our implementation achieves up to 98% resource savings with minimal accuracy loss, demonstrating its potential for scalable and resource-efficient multi-modal inference. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2504_07247 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Resource-efficient Inference with Foundation Model Programs Nie, Lunyiu Ding, Zhimin Yu, Kevin Cheung, Marco Jermaine, Chris Chaudhuri, Swarat Machine Learning The inference-time resource costs of large language and vision models present a growing challenge in production deployments. We propose the use of foundation model programs, i.e., programs that can invoke foundation models with varying resource costs and performance, as an approach to this problem. Specifically, we present a method that translates a task into a program, then learns a policy for resource allocation that, on each input, selects foundation model "backends" for each program module. The policy uses smaller, cheaper backends to handle simpler subtasks, while allowing more complex subtasks to leverage larger, more capable models. We evaluate the method on two new "streaming" visual question-answering tasks in which a system answers a question on a sequence of inputs, receiving ground-truth feedback after each answer. Compared to monolithic multi-modal models, our implementation achieves up to 98% resource savings with minimal accuracy loss, demonstrating its potential for scalable and resource-efficient multi-modal inference. |
| title | Resource-efficient Inference with Foundation Model Programs |
| topic | Machine Learning |
| url | https://arxiv.org/abs/2504.07247 |