Saved in:
| Main Authors: | , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2026
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2605.12036 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866913116989685760 |
|---|---|
| author | Li, Guojian Zhao, Zhixian Lin, Zhennan Hu, Jingbin Zhan, Qirui Cao, Yuang Xie, Pengyuan Xie, Chuan Liu, Jie Zhang, Qiang Fu, Zhonghua Xie, Lei |
| author_facet | Li, Guojian Zhao, Zhixian Lin, Zhennan Hu, Jingbin Zhan, Qirui Cao, Yuang Xie, Pengyuan Xie, Chuan Liu, Jie Zhang, Qiang Fu, Zhonghua Xie, Lei |
| contents | While speech Large Language Models (LLMs) excel at conventional tasks like basic speech recognition, they lack fine-grained, multi-dimensional perception. This deficiency is evident in their struggle to disentangle complex features like micro-acoustic cues, acoustic scenes, and paralinguistic signals. This resulting incomplete comprehension of real-world speech fundamentally bottlenecks the development of perceptive and empathetic next-generation speech systems. At its core, this persistent perceptual limitation primarily stems from three interacting factors: scarce high-quality expressive data, absent fine-grained modeling for multi-dimensional attributes, and reliance on restricted coverage, coarse-grained benchmarks. We address these challenges through three pillars: First, our robust data curation pipeline resolves complex acoustic environments and long-audio timestamp alignment challenges to extract a high-quality spontaneous speech corpus from audiovisual sources. Second, we construct FMSU-Bench, a pioneering benchmark covering 14 speech attribute dimensions to rigorously assess the fine-grained, multi-dimensional speech understanding capabilities of current models. Third, empowered by our curated corpus, we introduce FM-Speech. Driven by a decoupled attribute modeling and progressive curriculum fine-tuning framework, it substantially elevates fine-grained, multi-dimensional acoustic perception. Extensive evaluations on FMSU-Bench reveal that current speech LLMs still require significant improvement in multi-dimensional, fine-grained understanding. In contrast, FM-Speech substantially outperforms current open-source models, establishing a robust paradigm for real-world speech understanding. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2605_12036 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | Towards Fine-Grained Multi-Dimensional Speech Understanding: Data Pipeline, Benchmark, and Model Li, Guojian Zhao, Zhixian Lin, Zhennan Hu, Jingbin Zhan, Qirui Cao, Yuang Xie, Pengyuan Xie, Chuan Liu, Jie Zhang, Qiang Fu, Zhonghua Xie, Lei Audio and Speech Processing While speech Large Language Models (LLMs) excel at conventional tasks like basic speech recognition, they lack fine-grained, multi-dimensional perception. This deficiency is evident in their struggle to disentangle complex features like micro-acoustic cues, acoustic scenes, and paralinguistic signals. This resulting incomplete comprehension of real-world speech fundamentally bottlenecks the development of perceptive and empathetic next-generation speech systems. At its core, this persistent perceptual limitation primarily stems from three interacting factors: scarce high-quality expressive data, absent fine-grained modeling for multi-dimensional attributes, and reliance on restricted coverage, coarse-grained benchmarks. We address these challenges through three pillars: First, our robust data curation pipeline resolves complex acoustic environments and long-audio timestamp alignment challenges to extract a high-quality spontaneous speech corpus from audiovisual sources. Second, we construct FMSU-Bench, a pioneering benchmark covering 14 speech attribute dimensions to rigorously assess the fine-grained, multi-dimensional speech understanding capabilities of current models. Third, empowered by our curated corpus, we introduce FM-Speech. Driven by a decoupled attribute modeling and progressive curriculum fine-tuning framework, it substantially elevates fine-grained, multi-dimensional acoustic perception. Extensive evaluations on FMSU-Bench reveal that current speech LLMs still require significant improvement in multi-dimensional, fine-grained understanding. In contrast, FM-Speech substantially outperforms current open-source models, establishing a robust paradigm for real-world speech understanding. |
| title | Towards Fine-Grained Multi-Dimensional Speech Understanding: Data Pipeline, Benchmark, and Model |
| topic | Audio and Speech Processing |
| url | https://arxiv.org/abs/2605.12036 |