Multimodal Methods for Analyzing Learning and Training Environments: A Systematic Literature Review
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2024
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866914205954736128 |
|---|---|
| author | Cohn, Clayton Davalos, Eduardo Vatral, Caleb Fonteles, Joyce Horn Wang, Hanchen David Coursey, Austin Rayala, Surya S, Ashwin T Ma, Meiyi Biswas, Gautam |
| author_facet | Cohn, Clayton Davalos, Eduardo Vatral, Caleb Fonteles, Joyce Horn Wang, Hanchen David Coursey, Austin Rayala, Surya S, Ashwin T Ma, Meiyi Biswas, Gautam |
| contents | Recent technological advancements in multimodal machine learning--including the rise of large language models (LLMs)--have improved our ability to collect, process, and analyze diverse multimodal data such as speech, video, and eye gaze in learning and training contexts. While prior reviews have addressed individual components of the multimodal pipeline (e.g., conceptual models, data fusion), a comprehensive review of empirical methods in applied multimodal environments remains notably absent. This review addresses that, introducing a taxonomy and framework that capture both established practices and recent innovations driven by LLMs and generative AI. We identify five modality groups: Natural Language, Vision, Physiological Signals, Human-Centered Evidence, and Environment Logs. Our analysis reveals that integrating modalities enables richer insights into learner and trainee behaviors, revealing latent patterns often overlooked by unimodal approaches. However, persistent challenges in multimodal data collection and integration continue to hinder the adoption of these systems in real-time classroom settings. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2408_14491 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Multimodal Methods for Analyzing Learning and Training Environments: A Systematic Literature Review Cohn, Clayton Davalos, Eduardo Vatral, Caleb Fonteles, Joyce Horn Wang, Hanchen David Coursey, Austin Rayala, Surya S, Ashwin T Ma, Meiyi Biswas, Gautam Machine Learning Multimedia Recent technological advancements in multimodal machine learning--including the rise of large language models (LLMs)--have improved our ability to collect, process, and analyze diverse multimodal data such as speech, video, and eye gaze in learning and training contexts. While prior reviews have addressed individual components of the multimodal pipeline (e.g., conceptual models, data fusion), a comprehensive review of empirical methods in applied multimodal environments remains notably absent. This review addresses that, introducing a taxonomy and framework that capture both established practices and recent innovations driven by LLMs and generative AI. We identify five modality groups: Natural Language, Vision, Physiological Signals, Human-Centered Evidence, and Environment Logs. Our analysis reveals that integrating modalities enables richer insights into learner and trainee behaviors, revealing latent patterns often overlooked by unimodal approaches. However, persistent challenges in multimodal data collection and integration continue to hinder the adoption of these systems in real-time classroom settings. |
| title | Multimodal Methods for Analyzing Learning and Training Environments: A Systematic Literature Review |
| topic | Machine Learning Multimedia |
| url | https://arxiv.org/abs/2408.14491 |