Saved in:
| Main Authors: | , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2026
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2605.23975 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866911710210686976 |
|---|---|
| author | Quang, Trung Nguyen Won, Cheng Yi Lewis Pham, Minh Duc He, Yingxu Sun, Shuo Aw, Ai Ti |
| author_facet | Quang, Trung Nguyen Won, Cheng Yi Lewis Pham, Minh Duc He, Yingxu Sun, Shuo Aw, Ai Ti |
| contents | Audio large language models (Audio LLMs) exhibit systematic failures in transcribing code-switching speech despite strong multilingual capabilities. Focusing on English-Mandarin, we identify three failure modes: language omission, translation-instead-of-transcription, and hallucination. We apply Direct Preference Optimization (DPO) to align models, constructing preference pairs in which chosen responses preserve mixed-language content while rejected responses mimic failure patterns. Training three Audio LLMs on 100K pairs (570 hours), we observe consistent behavioral shifts: models learn to preserve language composition rather than translating when prompted for transcription. This alignment yields MER reductions up to 89.6% (in-distribution) and 20.0% (out-of-distribution). Our findings suggest DPO can effectively elicit correct code-switching transcription behavior from multilingual Audio LLMs. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2605_23975 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | Direct Preference Optimization for English-Mandarin Code-Switching Speech Recognition in Audio LLMs Quang, Trung Nguyen Won, Cheng Yi Lewis Pham, Minh Duc He, Yingxu Sun, Shuo Aw, Ai Ti Computation and Language Sound Audio large language models (Audio LLMs) exhibit systematic failures in transcribing code-switching speech despite strong multilingual capabilities. Focusing on English-Mandarin, we identify three failure modes: language omission, translation-instead-of-transcription, and hallucination. We apply Direct Preference Optimization (DPO) to align models, constructing preference pairs in which chosen responses preserve mixed-language content while rejected responses mimic failure patterns. Training three Audio LLMs on 100K pairs (570 hours), we observe consistent behavioral shifts: models learn to preserve language composition rather than translating when prompted for transcription. This alignment yields MER reductions up to 89.6% (in-distribution) and 20.0% (out-of-distribution). Our findings suggest DPO can effectively elicit correct code-switching transcription behavior from multilingual Audio LLMs. |
| title | Direct Preference Optimization for English-Mandarin Code-Switching Speech Recognition in Audio LLMs |
| topic | Computation and Language Sound |
| url | https://arxiv.org/abs/2605.23975 |