Using LLMs for Late Multimodal Sensor Fusion for Activity Recognition
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2025
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866911328739786752 |
|---|---|
| author | Demirel, Ilker Thakkar, Karan Elizalde, Benjamin Marques, Miquel Espi Sarathy, Aditya Bai, Yang Srinivas, Umamahesh Xu, Jiajie Ren, Shirley Narain, Jaya |
| author_facet | Demirel, Ilker Thakkar, Karan Elizalde, Benjamin Marques, Miquel Espi Sarathy, Aditya Bai, Yang Srinivas, Umamahesh Xu, Jiajie Ren, Shirley Narain, Jaya |
| contents | Sensor data streams provide valuable information around activities and context for downstream applications, though integrating complementary information can be challenging. We show that large language models (LLMs) can be used for late fusion for activity classification from audio and motion time series data. We curated a subset of data for diverse activity recognition across contexts (e.g., household activities, sports) from the Ego4D dataset. Evaluated LLMs achieved 12-class zero- and one-shot classification F1-scores significantly above chance, with no task-specific training. Zero-shot classification via LLM-based fusion from modality-specific models can enable multimodal temporal applications where there is limited aligned training data for learning a shared embedding space. Additionally, LLM-based fusion can enable model deploying without requiring additional memory and computation for targeted application-specific multimodal models. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2509_10729 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Using LLMs for Late Multimodal Sensor Fusion for Activity Recognition Demirel, Ilker Thakkar, Karan Elizalde, Benjamin Marques, Miquel Espi Sarathy, Aditya Bai, Yang Srinivas, Umamahesh Xu, Jiajie Ren, Shirley Narain, Jaya Machine Learning Sensor data streams provide valuable information around activities and context for downstream applications, though integrating complementary information can be challenging. We show that large language models (LLMs) can be used for late fusion for activity classification from audio and motion time series data. We curated a subset of data for diverse activity recognition across contexts (e.g., household activities, sports) from the Ego4D dataset. Evaluated LLMs achieved 12-class zero- and one-shot classification F1-scores significantly above chance, with no task-specific training. Zero-shot classification via LLM-based fusion from modality-specific models can enable multimodal temporal applications where there is limited aligned training data for learning a shared embedding space. Additionally, LLM-based fusion can enable model deploying without requiring additional memory and computation for targeted application-specific multimodal models. |
| title | Using LLMs for Late Multimodal Sensor Fusion for Activity Recognition |
| topic | Machine Learning |
| url | https://arxiv.org/abs/2509.10729 |