WhisperNER: Unified Open Named Entity and Speech Recognition
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866916884500185088 |
|---|---|
| author | Ayache, Gil Pirchi, Menachem Navon, Aviv Shamsian, Aviv Hetz, Gill Keshet, Joseph |
| author_facet | Ayache, Gil Pirchi, Menachem Navon, Aviv Shamsian, Aviv Hetz, Gill Keshet, Joseph |
| contents | Integrating named entity recognition (NER) with automatic speech recognition (ASR) can significantly enhance transcription accuracy and informativeness. In this paper, we introduce WhisperNER, a novel model that allows joint speech transcription and entity recognition. WhisperNER supports open-type NER, enabling recognition of diverse and evolving entities at inference. Building on recent advancements in open NER research, we augment a large synthetic dataset with synthetic speech samples. This allows us to train WhisperNER on a large number of examples with diverse NER tags. During training, the model is prompted with NER labels and optimized to output the transcribed utterance along with the corresponding tagged entities. To evaluate WhisperNER, we generate synthetic speech for commonly used NER benchmarks and annotate existing ASR datasets with open NER tags. Our experiments demonstrate that WhisperNER outperforms natural baselines on both out-of-domain open type NER and supervised finetuning. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2409_08107 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | WhisperNER: Unified Open Named Entity and Speech Recognition Ayache, Gil Pirchi, Menachem Navon, Aviv Shamsian, Aviv Hetz, Gill Keshet, Joseph Computation and Language Machine Learning Integrating named entity recognition (NER) with automatic speech recognition (ASR) can significantly enhance transcription accuracy and informativeness. In this paper, we introduce WhisperNER, a novel model that allows joint speech transcription and entity recognition. WhisperNER supports open-type NER, enabling recognition of diverse and evolving entities at inference. Building on recent advancements in open NER research, we augment a large synthetic dataset with synthetic speech samples. This allows us to train WhisperNER on a large number of examples with diverse NER tags. During training, the model is prompted with NER labels and optimized to output the transcribed utterance along with the corresponding tagged entities. To evaluate WhisperNER, we generate synthetic speech for commonly used NER benchmarks and annotate existing ASR datasets with open NER tags. Our experiments demonstrate that WhisperNER outperforms natural baselines on both out-of-domain open type NER and supervised finetuning. |
| title | WhisperNER: Unified Open Named Entity and Speech Recognition |
| topic | Computation and Language Machine Learning |
| url | https://arxiv.org/abs/2409.08107 |