Generative AI as a metacognitive agent: A comparative mixed-method study with human participants on ICF-mimicking exam performance
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2024
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866909196094537728 |
|---|---|
| author | Pavlovic, Jelena Krstic, Jugoslav Mitrovic, Luka Babic, Djordje Milosavljevic, Adrijana Nikolic, Milena Karaklic, Tijana Mitrovic, Tijana |
| author_facet | Pavlovic, Jelena Krstic, Jugoslav Mitrovic, Luka Babic, Djordje Milosavljevic, Adrijana Nikolic, Milena Karaklic, Tijana Mitrovic, Tijana |
| contents | This study investigates the metacognitive capabilities of Large Language Models relative to human metacognition in the context of the International Coaching Federation ICF mimicking exam, a situational judgment test related to coaching competencies. Using a mixed method approach, we assessed the metacognitive performance, including sensitivity, accuracy in probabilistic predictions, and bias, of human participants and five advanced LLMs (GPT-4, Claude-3-Opus 3, Mistral Large, Llama 3, and Gemini 1.5 Pro). The results indicate that LLMs outperformed humans across all metacognitive metrics, particularly in terms of reduced overconfidence, compared to humans. However, both LLMs and humans showed less adaptability in ambiguous scenarios, adhering closely to predefined decision frameworks. The study suggests that Generative AI can effectively engage in human-like metacognitive processing without conscious awareness. Implications of the study are discussed in relation to development of AI simulators that scaffold cognitive and metacognitive aspects of mastering coaching competencies. More broadly, implications of these results are discussed in relation to development of metacognitive modules that lead towards more autonomous and intuitive AI systems. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2405_05285 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Generative AI as a metacognitive agent: A comparative mixed-method study with human participants on ICF-mimicking exam performance Pavlovic, Jelena Krstic, Jugoslav Mitrovic, Luka Babic, Djordje Milosavljevic, Adrijana Nikolic, Milena Karaklic, Tijana Mitrovic, Tijana Human-Computer Interaction Artificial Intelligence This study investigates the metacognitive capabilities of Large Language Models relative to human metacognition in the context of the International Coaching Federation ICF mimicking exam, a situational judgment test related to coaching competencies. Using a mixed method approach, we assessed the metacognitive performance, including sensitivity, accuracy in probabilistic predictions, and bias, of human participants and five advanced LLMs (GPT-4, Claude-3-Opus 3, Mistral Large, Llama 3, and Gemini 1.5 Pro). The results indicate that LLMs outperformed humans across all metacognitive metrics, particularly in terms of reduced overconfidence, compared to humans. However, both LLMs and humans showed less adaptability in ambiguous scenarios, adhering closely to predefined decision frameworks. The study suggests that Generative AI can effectively engage in human-like metacognitive processing without conscious awareness. Implications of the study are discussed in relation to development of AI simulators that scaffold cognitive and metacognitive aspects of mastering coaching competencies. More broadly, implications of these results are discussed in relation to development of metacognitive modules that lead towards more autonomous and intuitive AI systems. |
| title | Generative AI as a metacognitive agent: A comparative mixed-method study with human participants on ICF-mimicking exam performance |
| topic | Human-Computer Interaction Artificial Intelligence |
| url | https://arxiv.org/abs/2405.05285 |