Saved in:
| Main Authors: | , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2408.14972 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866916371196018688 |
|---|---|
| author | Chan, Chi-Min Yu, Jianxuan Chen, Weize Jiang, Chunyang Liu, Xinyu Shi, Weijie Liu, Zhiyuan Xue, Wei Guo, Yike |
| author_facet | Chan, Chi-Min Yu, Jianxuan Chen, Weize Jiang, Chunyang Liu, Xinyu Shi, Weijie Liu, Zhiyuan Xue, Wei Guo, Yike |
| contents | The rapid advancement of large language models (LLMs) has led to the rise of LLM-based agents. Recent research shows that multi-agent systems (MAS), where each agent plays a specific role, can outperform individual LLMs. However, configuring an MAS for a task remains challenging, with performance only observable post-execution. Inspired by scaling laws in LLM development, we investigate whether MAS performance can be predicted beforehand. We introduce AgentMonitor, a framework that integrates at the agent level to capture inputs and outputs, transforming them into statistics for training a regression model to predict task performance. Additionally, it can further apply real-time corrections to address security risks posed by malicious agents, mitigating negative impacts and enhancing MAS security. Experiments demonstrate that an XGBoost model achieves a Spearman correlation of 0.89 in-domain and 0.58 in more challenging scenarios. Furthermore, using AgentMonitor reduces harmful content by 6.2% and increases helpful content by 1.8% on average, enhancing safety and reliability. Code is available at \url{https://github.com/chanchimin/AgentMonitor}. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2408_14972 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | AgentMonitor: A Plug-and-Play Framework for Predictive and Secure Multi-Agent Systems Chan, Chi-Min Yu, Jianxuan Chen, Weize Jiang, Chunyang Liu, Xinyu Shi, Weijie Liu, Zhiyuan Xue, Wei Guo, Yike Computation and Language The rapid advancement of large language models (LLMs) has led to the rise of LLM-based agents. Recent research shows that multi-agent systems (MAS), where each agent plays a specific role, can outperform individual LLMs. However, configuring an MAS for a task remains challenging, with performance only observable post-execution. Inspired by scaling laws in LLM development, we investigate whether MAS performance can be predicted beforehand. We introduce AgentMonitor, a framework that integrates at the agent level to capture inputs and outputs, transforming them into statistics for training a regression model to predict task performance. Additionally, it can further apply real-time corrections to address security risks posed by malicious agents, mitigating negative impacts and enhancing MAS security. Experiments demonstrate that an XGBoost model achieves a Spearman correlation of 0.89 in-domain and 0.58 in more challenging scenarios. Furthermore, using AgentMonitor reduces harmful content by 6.2% and increases helpful content by 1.8% on average, enhancing safety and reliability. Code is available at \url{https://github.com/chanchimin/AgentMonitor}. |
| title | AgentMonitor: A Plug-and-Play Framework for Predictive and Secure Multi-Agent Systems |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2408.14972 |