SensorBench: Benchmarking LLMs in Coding-Based Sensor Processing
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2024
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866910898670534656 |
|---|---|
| author | Quan, Pengrui Ouyang, Xiaomin Jeyakumar, Jeya Vikranth Wang, Ziqi Xing, Yang Srivastava, Mani |
| author_facet | Quan, Pengrui Ouyang, Xiaomin Jeyakumar, Jeya Vikranth Wang, Ziqi Xing, Yang Srivastava, Mani |
| contents | Effective processing, interpretation, and management of sensor data have emerged as a critical component of cyber-physical systems. Traditionally, processing sensor data requires profound theoretical knowledge and proficiency in signal-processing tools. However, recent works show that Large Language Models (LLMs) have promising capabilities in processing sensory data, suggesting their potential as copilots for developing sensing systems.
To explore this potential, we construct a comprehensive benchmark, SensorBench, to establish a quantifiable objective. The benchmark incorporates diverse real-world sensor datasets for various tasks. The results show that while LLMs exhibit considerable proficiency in simpler tasks, they face inherent challenges in processing compositional tasks with parameter selections compared to engineering experts. Additionally, we investigate four prompting strategies for sensor processing and show that self-verification can outperform all other baselines in 48% of tasks. Our study provides a comprehensive benchmark and prompting analysis for future developments, paving the way toward an LLM-based sensor processing copilot. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2410_10741 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | SensorBench: Benchmarking LLMs in Coding-Based Sensor Processing Quan, Pengrui Ouyang, Xiaomin Jeyakumar, Jeya Vikranth Wang, Ziqi Xing, Yang Srivastava, Mani Artificial Intelligence Machine Learning Signal Processing Effective processing, interpretation, and management of sensor data have emerged as a critical component of cyber-physical systems. Traditionally, processing sensor data requires profound theoretical knowledge and proficiency in signal-processing tools. However, recent works show that Large Language Models (LLMs) have promising capabilities in processing sensory data, suggesting their potential as copilots for developing sensing systems. To explore this potential, we construct a comprehensive benchmark, SensorBench, to establish a quantifiable objective. The benchmark incorporates diverse real-world sensor datasets for various tasks. The results show that while LLMs exhibit considerable proficiency in simpler tasks, they face inherent challenges in processing compositional tasks with parameter selections compared to engineering experts. Additionally, we investigate four prompting strategies for sensor processing and show that self-verification can outperform all other baselines in 48% of tasks. Our study provides a comprehensive benchmark and prompting analysis for future developments, paving the way toward an LLM-based sensor processing copilot. |
| title | SensorBench: Benchmarking LLMs in Coding-Based Sensor Processing |
| topic | Artificial Intelligence Machine Learning Signal Processing |
| url | https://arxiv.org/abs/2410.10741 |