Saved in:
| Main Authors: | , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2511.09915 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866912708698308608 |
|---|---|
| author | Ma, Zhiming Gan, Shiyu Zhao, Junhao Li, Xianming Pan, Qingyun Wang, Peidong Pan, Mingjun Mo, Yuhao Cheng, Jiajie Chen, Chengxin Cao, Zhonglun Liu, Chonghan Cheng, Shi |
| author_facet | Ma, Zhiming Gan, Shiyu Zhao, Junhao Li, Xianming Pan, Qingyun Wang, Peidong Pan, Mingjun Mo, Yuhao Cheng, Jiajie Chen, Chengxin Cao, Zhonglun Liu, Chonghan Cheng, Shi |
| contents | Hearing-impaired individuals often face significant barriers in daily communication due to the inherent challenges of producing clear speech. To address this, we introduce the Omni-Model paradigm into assistive technology and present HI-TransPA, an instruction-driven audio-visual personal assistant. The model fuses indistinct speech with lip dynamics, enabling both translation and dialogue within a single multimodal framework. To address the distinctive pronunciation patterns of hearing-impaired speech and the limited adaptability of existing models, we develop a multimodal preprocessing and curation pipeline that detects facial landmarks, stabilizes the lip region, and quantitatively evaluates sample quality. These quality scores guide a curriculum learning strategy that first trains on clean, high-confidence samples and progressively incorporates harder cases to strengthen model robustness. Architecturally, we employs a novel unified 3D-Resampler to efficiently encode the lip dynamics, which is critical for accurate interpretation. Experiments on purpose-built HI-Dialogue dataset show that HI-TransPA achieves state-of-the-art performance in both literal accuracy and semantic fidelity. Our work establishes a foundation for applying Omni-Models to assistive communication technology, providing an end-to-end modeling framework and essential processing tools for future research. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2511_09915 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | HI-TransPA: Hearing Impairments Translation Personal Assistant Ma, Zhiming Gan, Shiyu Zhao, Junhao Li, Xianming Pan, Qingyun Wang, Peidong Pan, Mingjun Mo, Yuhao Cheng, Jiajie Chen, Chengxin Cao, Zhonglun Liu, Chonghan Cheng, Shi Computation and Language Multimedia Sound Hearing-impaired individuals often face significant barriers in daily communication due to the inherent challenges of producing clear speech. To address this, we introduce the Omni-Model paradigm into assistive technology and present HI-TransPA, an instruction-driven audio-visual personal assistant. The model fuses indistinct speech with lip dynamics, enabling both translation and dialogue within a single multimodal framework. To address the distinctive pronunciation patterns of hearing-impaired speech and the limited adaptability of existing models, we develop a multimodal preprocessing and curation pipeline that detects facial landmarks, stabilizes the lip region, and quantitatively evaluates sample quality. These quality scores guide a curriculum learning strategy that first trains on clean, high-confidence samples and progressively incorporates harder cases to strengthen model robustness. Architecturally, we employs a novel unified 3D-Resampler to efficiently encode the lip dynamics, which is critical for accurate interpretation. Experiments on purpose-built HI-Dialogue dataset show that HI-TransPA achieves state-of-the-art performance in both literal accuracy and semantic fidelity. Our work establishes a foundation for applying Omni-Models to assistive communication technology, providing an end-to-end modeling framework and essential processing tools for future research. |
| title | HI-TransPA: Hearing Impairments Translation Personal Assistant |
| topic | Computation and Language Multimedia Sound |
| url | https://arxiv.org/abs/2511.09915 |