Saved in:
| Main Authors: | , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2508.12666 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866909740299190272 |
|---|---|
| author | Mitrofanov, Anton Novoselov, Sergei Prisyach, Tatiana Marchevskiy, Vladislav Karelin, Arseniy Khmelev, Nikita Dutov, Dmitry Malykh, Stepan Agafonov, Igor Nikitin, Aleksandr Petrov, Oleg |
| author_facet | Mitrofanov, Anton Novoselov, Sergei Prisyach, Tatiana Marchevskiy, Vladislav Karelin, Arseniy Khmelev, Nikita Dutov, Dmitry Malykh, Stepan Agafonov, Igor Nikitin, Aleksandr Petrov, Oleg |
| contents | The recent revolutionary progress in text-based large language models (LLMs) has contributed to the growth of interest in extending capabilities of such models to multimodal perception and understanding tasks. Hearing is an essential capability that is highly desired to be integrated into LLMs. However, effective integrating listening capabilities into LLMs is a significant challenge lying in generalizing complex auditory tasks across speech and sounds. To address these issues, we introduce Cryfish, our version of auditory-capable LLM. The model integrates WavLM audio-encoder features into Qwen2 model using a transformer-based connector. Cryfish is adapted to various auditory tasks through a specialized training strategy. We evaluate the model on the new Dynamic SUPERB Phase-2 comprehensive multitask benchmark specifically designed for auditory-capable models. The paper presents an in-depth analysis and detailed comparison of Cryfish with the publicly available models. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2508_12666 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Cryfish: On deep audio analysis with Large Language Models Mitrofanov, Anton Novoselov, Sergei Prisyach, Tatiana Marchevskiy, Vladislav Karelin, Arseniy Khmelev, Nikita Dutov, Dmitry Malykh, Stepan Agafonov, Igor Nikitin, Aleksandr Petrov, Oleg Audio and Speech Processing The recent revolutionary progress in text-based large language models (LLMs) has contributed to the growth of interest in extending capabilities of such models to multimodal perception and understanding tasks. Hearing is an essential capability that is highly desired to be integrated into LLMs. However, effective integrating listening capabilities into LLMs is a significant challenge lying in generalizing complex auditory tasks across speech and sounds. To address these issues, we introduce Cryfish, our version of auditory-capable LLM. The model integrates WavLM audio-encoder features into Qwen2 model using a transformer-based connector. Cryfish is adapted to various auditory tasks through a specialized training strategy. We evaluate the model on the new Dynamic SUPERB Phase-2 comprehensive multitask benchmark specifically designed for auditory-capable models. The paper presents an in-depth analysis and detailed comparison of Cryfish with the publicly available models. |
| title | Cryfish: On deep audio analysis with Large Language Models |
| topic | Audio and Speech Processing |
| url | https://arxiv.org/abs/2508.12666 |