MALT: Improving Reasoning with Multi-Agent LLM Training
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866918154912923648 |
|---|---|
| author | Motwani, Sumeet Ramesh Smith, Chandler Das, Rocktim Jyoti Rafailov, Rafael Laptev, Ivan Torr, Philip H. S. Pizzati, Fabio Clark, Ronald de Witt, Christian Schroeder |
| author_facet | Motwani, Sumeet Ramesh Smith, Chandler Das, Rocktim Jyoti Rafailov, Rafael Laptev, Ivan Torr, Philip H. S. Pizzati, Fabio Clark, Ronald de Witt, Christian Schroeder |
| contents | Large Language Models (LLMs) often produce answers with a single chain-of-thought, which restricts their ability to explore reasoning paths or self-correct flawed outputs in complex tasks. In this paper, we introduce MALT (Multi-Agent LLM Training), a novel post-training strategy that divides the reasoning process into generation, verification, and refinement steps using a sequential pipeline of heterogeneous agents. During data generation, each agent is repeatedly sampled to form a multi-agent search tree, where final outputs are graded against ground-truth data. We then apply value iteration to propagate reward signals back to each role-conditioned model, automatically producing multi-agent post-training data without human or teacher-model supervision. Our off-policy approach allows each agent to specialize by learning from correct and incorrect trajectories, ultimately improving the end-to-end reasoning chain. On MATH, GSM8K, and CSQA, MALT surpasses the same baseline LLM with a relative improvement of 15.66%, 7.42%, and 9.40% respectively, making it an important advance towards multi-agent cooperative training. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2412_01928 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | MALT: Improving Reasoning with Multi-Agent LLM Training Motwani, Sumeet Ramesh Smith, Chandler Das, Rocktim Jyoti Rafailov, Rafael Laptev, Ivan Torr, Philip H. S. Pizzati, Fabio Clark, Ronald de Witt, Christian Schroeder Machine Learning Artificial Intelligence Large Language Models (LLMs) often produce answers with a single chain-of-thought, which restricts their ability to explore reasoning paths or self-correct flawed outputs in complex tasks. In this paper, we introduce MALT (Multi-Agent LLM Training), a novel post-training strategy that divides the reasoning process into generation, verification, and refinement steps using a sequential pipeline of heterogeneous agents. During data generation, each agent is repeatedly sampled to form a multi-agent search tree, where final outputs are graded against ground-truth data. We then apply value iteration to propagate reward signals back to each role-conditioned model, automatically producing multi-agent post-training data without human or teacher-model supervision. Our off-policy approach allows each agent to specialize by learning from correct and incorrect trajectories, ultimately improving the end-to-end reasoning chain. On MATH, GSM8K, and CSQA, MALT surpasses the same baseline LLM with a relative improvement of 15.66%, 7.42%, and 9.40% respectively, making it an important advance towards multi-agent cooperative training. |
| title | MALT: Improving Reasoning with Multi-Agent LLM Training |
| topic | Machine Learning Artificial Intelligence |
| url | https://arxiv.org/abs/2412.01928 |