Cover Image

Data-Centric Improvements for Enhancing Multi-Modal Understanding in Spoken Conversation Modeling

Fuente: arXiv

Saved in:

Bibliographic Details
Main Authors:	Chen, Maximillian, Sun, Ruoxi, Arık, Sercan Ö.
Format:	Preprint
Published:	2024
Subjects:	Computation and Language Artificial Intelligence Sound Audio and Speech Processing
Online Access:	Acceder al recurso
Tags:	Add Tag No Tags, Be the first to tag this record!

Similar Items

Evaluating and Improving Continual Learning in Spoken Language Understanding
by: Yang, Muqiao, et al.
Published: (2024)

Data Augmentation for Spoken Grammatical Error Correction
by: Karanasou, Penny, et al.
Published: (2025)

MULTI-Bench: A Multi-Turn Interactive Benchmark for Assessing Emotional Intelligence ability of Spoken Dialogue Models
by: Deng, Yayue, et al.
Published: (2025)

Retrieval Augmented End-to-End Spoken Dialog Models
by: Wang, Mingqiu, et al.
Published: (2024)

GOAT-SLM: A Spoken Language Model with Paralinguistic and Speaker Characteristic Awareness
by: Chen, Hongjie, et al.
Published: (2025)

UniSLU: Unified Spoken Language Understanding from Heterogeneous Cross-Task Datasets
by: Sheng, Zhichao, et al.
Published: (2025)

TELEVAL: A Dynamic Benchmark Designed for Spoken Language Models in Chinese Interactive Scenarios
by: Li, Zehan, et al.
Published: (2025)

VStyle: A Benchmark for Voice Style Adaptation with Spoken Instructions
by: Zhan, Jun, et al.
Published: (2025)

NTPP: Generative Speech Language Modeling for Dual-Channel Spoken Dialogue via Next-Token-Pair Prediction
by: Wang, Qichao, et al.
Published: (2025)

An Efficient Self-Learning Framework For Interactive Spoken Dialog Systems
by: Tulsiani, Hitesh, et al.
Published: (2024)

MUGEN: Evaluating and Improving Multi-audio Understanding of Large Audio-Language Models
by: Yang, Chih-Kai, et al.
Published: (2026)

On the Evaluation of Speech Foundation Models for Spoken Language Understanding
by: Arora, Siddhant, et al.
Published: (2024)

Multi-Modal Automatic Prosody Annotation with Contrastive Pretraining of SSWP
by: Zhong, Jinzuomu, et al.
Published: (2023)

MMSU: A Massive Multi-task Spoken Language Understanding and Reasoning Benchmark
by: Wang, Dingdong, et al.
Published: (2025)

LLaMA-Omni2: LLM-based Real-time Spoken Chatbot with Autoregressive Streaming Speech Synthesis
by: Fang, Qingkai, et al.
Published: (2025)

MECAT: A Multi-Experts Constructed Benchmark for Fine-Grained Audio Understanding Tasks
by: Niu, Yadong, et al.
Published: (2025)

OmniFlatten: An End-to-end GPT Model for Seamless Voice Conversation
by: Zhang, Qinglin, et al.
Published: (2024)

Dual Information Speech Language Models for Emotional Conversations
by: Wang, Chun, et al.
Published: (2025)

MMAU: A Massive Multi-Task Audio Understanding and Reasoning Benchmark
by: Sakshi, S, et al.
Published: (2024)

Low-Rank and Sparse Model Merging for Multi-Lingual Speech Recognition and Translation
by: Zhao, Qiuming, et al.
Published: (2025)

A Variational Framework for Improving Naturalness in Generative Spoken Language Models
by: Chen, Li-Wei, et al.
Published: (2025)

Enhancing Low-Resource Language and Instruction Following Capabilities of Audio Language Models
by: Manakul, Potsawee, et al.
Published: (2024)

Device-Directed Speech Detection for Follow-up Conversations Using Large Language Models
by: Ognjen, et al.
Published: (2024)

MIST: Multimodal Interactive Speech-based Tool-calling Conversational Assistants for Smart Homes
by: Chen, Maximillian, et al.
Published: (2026)

SecoustiCodec: Cross-Modal Aligned Streaming Single-Codecbook Speech Codec
by: Qiang, Chunyu, et al.
Published: (2025)

Classifier-Guided Captioning Across Modalities
by: Shaulov, Ariel, et al.
Published: (2025)

SALMONN-omni: A Codec-free LLM for Full-duplex Speech Understanding and Generation
by: Yu, Wenyi, et al.
Published: (2024)

SightSound-R1: Cross-Modal Reasoning Distillation from Vision to Audio Language Models
by: Wang, Qiaolin, et al.
Published: (2025)

Spoken Language Intelligence of Large Language Models for Language Learning
by: Peng, Linkai, et al.
Published: (2023)

TSPC: A Two-Stage Phoneme-Centric Architecture for code-switching Vietnamese-English Speech Recognition
by: Anh, Tran Nguyen, et al.
Published: (2025)

Roadmap towards Superhuman Speech Understanding using Large Language Models
by: Bu, Fan, et al.
Published: (2024)

Benchmarking Open-ended Audio Dialogue Understanding for Large Audio-Language Models
by: Gao, Kuofeng, et al.
Published: (2024)

Extending Audio Context for Long-Form Understanding in Large Audio-Language Models
by: Chaichana, Yuatyong, et al.
Published: (2025)

DeepResonance: Enhancing Multimodal Music Understanding via Music-centric Multi-way Instruction Tuning
by: Mao, Zhuoyuan, et al.
Published: (2025)

Analyzing Mitigation Strategies for Catastrophic Forgetting in End-to-End Training of Spoken Language Models
by: Hsiao, Chi-Yuan, et al.
Published: (2025)

VQ-CTAP: Cross-Modal Fine-Grained Sequence Representation Learning for Speech Processing
by: Qiang, Chunyu, et al.
Published: (2024)

A Self-Refining Framework for Enhancing ASR Using TTS-Synthesized Data
by: Chou, Cheng-Kang, et al.
Published: (2025)

VoxEval: Benchmarking the Knowledge Understanding Capabilities of End-to-End Spoken Language Models
by: Cui, Wenqian, et al.
Published: (2025)

KidSpeak: A General Multi-purpose LLM for Kids' Speech Recognition and Screening
by: Sharma, Rohan, et al.
Published: (2025)

KoALa-Bench: Evaluating Large Audio Language Models on Korean Speech Understanding and Faithfulness
by: Kim, Jinyoung, et al.
Published: (2026)