Soft Measures for Extracting Causal Collective Intelligence
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2024
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866908389994397696 |
|---|---|
| author | Berijanian, Maryam Dork, Spencer Singh, Kuldeep Millikan, Michael Riley Riggs, Ashlin Swaminathan, Aadarsh Gibbs, Sarah L. Friedman, Scott E. Brugnone, Nathan |
| author_facet | Berijanian, Maryam Dork, Spencer Singh, Kuldeep Millikan, Michael Riley Riggs, Ashlin Swaminathan, Aadarsh Gibbs, Sarah L. Friedman, Scott E. Brugnone, Nathan |
| contents | Understanding and modeling collective intelligence is essential for addressing complex social systems. Directed graphs called fuzzy cognitive maps (FCMs) offer a powerful tool for encoding causal mental models, but extracting high-integrity FCMs from text is challenging. This study presents an approach using large language models (LLMs) to automate FCM extraction. We introduce novel graph-based similarity measures and evaluate them by correlating their outputs with human judgments through the Elo rating system. Results show positive correlations with human evaluations, but even the best-performing measure exhibits limitations in capturing FCM nuances. Fine-tuning LLMs improves performance, but existing measures still fall short. This study highlights the need for soft similarity measures tailored to FCM extraction, advancing collective intelligence modeling with NLP. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2409_18911 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Soft Measures for Extracting Causal Collective Intelligence Berijanian, Maryam Dork, Spencer Singh, Kuldeep Millikan, Michael Riley Riggs, Ashlin Swaminathan, Aadarsh Gibbs, Sarah L. Friedman, Scott E. Brugnone, Nathan Computation and Language Artificial Intelligence Computers and Society Social and Information Networks Understanding and modeling collective intelligence is essential for addressing complex social systems. Directed graphs called fuzzy cognitive maps (FCMs) offer a powerful tool for encoding causal mental models, but extracting high-integrity FCMs from text is challenging. This study presents an approach using large language models (LLMs) to automate FCM extraction. We introduce novel graph-based similarity measures and evaluate them by correlating their outputs with human judgments through the Elo rating system. Results show positive correlations with human evaluations, but even the best-performing measure exhibits limitations in capturing FCM nuances. Fine-tuning LLMs improves performance, but existing measures still fall short. This study highlights the need for soft similarity measures tailored to FCM extraction, advancing collective intelligence modeling with NLP. |
| title | Soft Measures for Extracting Causal Collective Intelligence |
| topic | Computation and Language Artificial Intelligence Computers and Society Social and Information Networks |
| url | https://arxiv.org/abs/2409.18911 |