Can We Trust AI Agents? A Case Study of an LLM-Based Multi-Agent System for Ethical AI
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2024
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866909611562369024 |
|---|---|
| author | de Cerqueira, José Antonio Siqueira Agbese, Mamia Rousi, Rebekah Xi, Nannan Hamari, Juho Abrahamsson, Pekka |
| author_facet | de Cerqueira, José Antonio Siqueira Agbese, Mamia Rousi, Rebekah Xi, Nannan Hamari, Juho Abrahamsson, Pekka |
| contents | AI-based systems, including Large Language Models (LLM), impact millions by supporting diverse tasks but face issues like misinformation, bias, and misuse. AI ethics is crucial as new technologies and concerns emerge, but objective, practical guidance remains debated. This study examines the use of LLMs for AI ethics in practice, assessing how LLM trustworthiness-enhancing techniques affect software development in this context. Using the Design Science Research (DSR) method, we identify techniques for LLM trustworthiness: multi-agents, distinct roles, structured communication, and multiple rounds of debate. We design a multi-agent prototype LLM-MAS, where agents engage in structured discussions on real-world AI ethics issues from the AI Incident Database. We evaluate the prototype across three case scenarios using thematic analysis, hierarchical clustering, comparative (baseline) studies, and running source code. The system generates approximately 2,000 lines of code per case, compared to only 80 lines in baseline trials. Discussions reveal terms like bias detection, transparency, accountability, user consent, GDPR compliance, fairness evaluation, and EU AI Act compliance, showing this prototype ability to generate extensive source code and documentation addressing often overlooked AI ethics issues. However, practical challenges in source code integration and dependency management may limit its use by practitioners. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2411_08881 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Can We Trust AI Agents? A Case Study of an LLM-Based Multi-Agent System for Ethical AI de Cerqueira, José Antonio Siqueira Agbese, Mamia Rousi, Rebekah Xi, Nannan Hamari, Juho Abrahamsson, Pekka Computers and Society Artificial Intelligence I.2.0; K.6.3 AI-based systems, including Large Language Models (LLM), impact millions by supporting diverse tasks but face issues like misinformation, bias, and misuse. AI ethics is crucial as new technologies and concerns emerge, but objective, practical guidance remains debated. This study examines the use of LLMs for AI ethics in practice, assessing how LLM trustworthiness-enhancing techniques affect software development in this context. Using the Design Science Research (DSR) method, we identify techniques for LLM trustworthiness: multi-agents, distinct roles, structured communication, and multiple rounds of debate. We design a multi-agent prototype LLM-MAS, where agents engage in structured discussions on real-world AI ethics issues from the AI Incident Database. We evaluate the prototype across three case scenarios using thematic analysis, hierarchical clustering, comparative (baseline) studies, and running source code. The system generates approximately 2,000 lines of code per case, compared to only 80 lines in baseline trials. Discussions reveal terms like bias detection, transparency, accountability, user consent, GDPR compliance, fairness evaluation, and EU AI Act compliance, showing this prototype ability to generate extensive source code and documentation addressing often overlooked AI ethics issues. However, practical challenges in source code integration and dependency management may limit its use by practitioners. |
| title | Can We Trust AI Agents? A Case Study of an LLM-Based Multi-Agent System for Ethical AI |
| topic | Computers and Society Artificial Intelligence I.2.0; K.6.3 |
| url | https://arxiv.org/abs/2411.08881 |