Computational Economics in Large Language Models: Exploring Model Behavior and Incentive Design under Resource Constraints
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2025
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866911342110179328 |
|---|---|
| author | Reddy, Sandeep Khan, Kabir Patil, Rohit Chakraborty, Ananya Khan, Faizan A. Kulkarni, Swati Verma, Arjun Singh, Neha |
| author_facet | Reddy, Sandeep Khan, Kabir Patil, Rohit Chakraborty, Ananya Khan, Faizan A. Kulkarni, Swati Verma, Arjun Singh, Neha |
| contents | Large language models (LLMs) are limited by substantial computational cost. We introduce a "computational economics" framework that treats an LLM as an internal economy of resource-constrained agents (attention heads and neuron blocks) that must allocate scarce computation to maximize task utility. First, we show empirically that when computation is scarce, standard LLMs reallocate attention toward high-value tokens while preserving accuracy. Building on this observation, we propose an incentive-driven training paradigm that augments the task loss with a differentiable computation cost term, encouraging sparse and efficient activations. On GLUE (MNLI, STS-B, CoLA) and WikiText-103, the method yields a family of models that trace a Pareto frontier and consistently dominate post-hoc pruning; for a similar accuracy we obtain roughly a forty percent reduction in FLOPS and lower latency, together with more interpretable attention patterns. These results indicate that economic principles offer a principled route to designing efficient, adaptive, and more transparent LLMs under strict resource constraints. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2508_10426 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Computational Economics in Large Language Models: Exploring Model Behavior and Incentive Design under Resource Constraints Reddy, Sandeep Khan, Kabir Patil, Rohit Chakraborty, Ananya Khan, Faizan A. Kulkarni, Swati Verma, Arjun Singh, Neha Computation and Language I.2.6; I.2.7; I.5.1 Large language models (LLMs) are limited by substantial computational cost. We introduce a "computational economics" framework that treats an LLM as an internal economy of resource-constrained agents (attention heads and neuron blocks) that must allocate scarce computation to maximize task utility. First, we show empirically that when computation is scarce, standard LLMs reallocate attention toward high-value tokens while preserving accuracy. Building on this observation, we propose an incentive-driven training paradigm that augments the task loss with a differentiable computation cost term, encouraging sparse and efficient activations. On GLUE (MNLI, STS-B, CoLA) and WikiText-103, the method yields a family of models that trace a Pareto frontier and consistently dominate post-hoc pruning; for a similar accuracy we obtain roughly a forty percent reduction in FLOPS and lower latency, together with more interpretable attention patterns. These results indicate that economic principles offer a principled route to designing efficient, adaptive, and more transparent LLMs under strict resource constraints. |
| title | Computational Economics in Large Language Models: Exploring Model Behavior and Incentive Design under Resource Constraints |
| topic | Computation and Language I.2.6; I.2.7; I.5.1 |
| url | https://arxiv.org/abs/2508.10426 |