Do What I Say: A Spoken Prompt Dataset for Instruction-Following
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2026
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866913074856853504 |
|---|---|
| author | Züfle, Maike Papi, Sara Retkowski, Fabian Mazurek, Szymon Kasztelnik, Marek Waibel, Alexander Bentivogli, Luisa Niehues, Jan |
| author_facet | Züfle, Maike Papi, Sara Retkowski, Fabian Mazurek, Szymon Kasztelnik, Marek Waibel, Alexander Bentivogli, Luisa Niehues, Jan |
| contents | Speech Large Language Models (SLLMs) have rapidly expanded, supporting a wide range of tasks. These models are typically evaluated using text prompts, which may not reflect real-world scenarios where users interact with speech. To address this gap, we introduce DoWhatISay (DOWIS), a multilingual dataset of human-recorded spoken and written prompts designed to pair with any existing benchmark for realistic evaluation of SLLMs under spoken instruction conditions. Spanning 9 tasks and 11 languages, it provides 10 prompt variants per task-language pair, across five styles. Using DOWIS, we benchmark state-of-the-art SLLMs, analyzing the interplay between prompt modality, style, language, and task type. Results show that text prompts consistently outperform spoken prompts, particularly for low-resource and cross-lingual settings. Only for tasks with speech output, spoken prompts do close the gap, highlighting the need for speech-based prompting in SLLM evaluation. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2603_09881 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | Do What I Say: A Spoken Prompt Dataset for Instruction-Following Züfle, Maike Papi, Sara Retkowski, Fabian Mazurek, Szymon Kasztelnik, Marek Waibel, Alexander Bentivogli, Luisa Niehues, Jan Computation and Language Speech Large Language Models (SLLMs) have rapidly expanded, supporting a wide range of tasks. These models are typically evaluated using text prompts, which may not reflect real-world scenarios where users interact with speech. To address this gap, we introduce DoWhatISay (DOWIS), a multilingual dataset of human-recorded spoken and written prompts designed to pair with any existing benchmark for realistic evaluation of SLLMs under spoken instruction conditions. Spanning 9 tasks and 11 languages, it provides 10 prompt variants per task-language pair, across five styles. Using DOWIS, we benchmark state-of-the-art SLLMs, analyzing the interplay between prompt modality, style, language, and task type. Results show that text prompts consistently outperform spoken prompts, particularly for low-resource and cross-lingual settings. Only for tasks with speech output, spoken prompts do close the gap, highlighting the need for speech-based prompting in SLLM evaluation. |
| title | Do What I Say: A Spoken Prompt Dataset for Instruction-Following |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2603.09881 |